task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-5z868-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-5z868-kubeconfig [get-kubeconfig] Found admin password secret: cluster-5z868-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-5z868-password [get-kubeconfig] API Server URL: https://a692ccb1eeed14afa90aadf4cc55a129-e6f7d1a45d273665.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.121837d2-f4fb-4805-b181-429475e54e2f.prod.konfluxeaas.com [clone-repo] dependabot/uv/python/test_resources/graph/success_200_isvc/python-multipart-0.0.26 [clone-repo] https://github.com/opendatahub-io/kserve [clone-repo] Cloning into '/workspace/source'... [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:1f655ab77a6d99cdde8a7f1f5e600af19cd3c90848f1adcb2e5105b44606fd11 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:1f655ab77a6d99cdde8a7f1f5e600af19cd3c90848f1adcb2e5105b44606fd11 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:2311ee42fdc1c8c2afc912b753776069bc3cfdc60b23f3f56b12f51c04dc6b51 [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:2311ee42fdc1c8c2afc912b753776069bc3cfdc60b23f3f56b12f51c04dc6b51 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:013b59be0bfa982c0e31bad4b53ad46f4b1067e5dae5be631dfd76f63f738c54 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:013b59be0bfa982c0e31bad4b53ad46f4b1067e5dae5be631dfd76f63f738c54 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:48bcc0e7da25ba7560a82df7216c0ad464b268d851c665b85b9195165f27b89d [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:48bcc0e7da25ba7560a82df7216c0ad464b268d851c665b85b9195165f27b89d [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:949b95fa3542c73dde7e415bf17e81749369919a83db635b8a7c415bb4e2e663 [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:949b95fa3542c73dde7e415bf17e81749369919a83db635b8a7c415bb4e2e663 [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu' 2 llm-d [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1417 [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:2311ee42fdc1c8c2afc912b753776069bc3cfdc60b23f3f56b12f51c04dc6b51 [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:949b95fa3542c73dde7e415bf17e81749369919a83db635b8a7c415bb4e2e663 [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:1f655ab77a6d99cdde8a7f1f5e600af19cd3c90848f1adcb2e5105b44606fd11 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:013b59be0bfa982c0e31bad4b53ad46f4b1067e5dae5be631dfd76f63f738c54 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:48bcc0e7da25ba7560a82df7216c0ad464b268d851c665b85b9195165f27b89d [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1417 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1417 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 259 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading cryptography (4.3MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Building pyasn==1.6.2 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Built pyasn==1.6.2 [e2e-llm-inference-service] Prepared 100 packages in 3.45s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 100 packages in 264ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.13.3 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.1.8 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==46.0.5 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.121.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.18.0rc1 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.10.16 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn==1.6.2 [e2e-llm-inference-service] + pyasn1==0.6.1 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.12.1 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.22 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==0.49.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.6.2 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 52ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.18.1-2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.18.1-2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.18.1-2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.18.1-2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.18.1-2 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.18.1-2 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-ffbb595cb-rlg54 1/1 Running 0 42s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-ffbb595cb-rlg54 condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-7c9f485588-ff4wr 1/1 Running 0 47s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-7c9f485588-ff4wr condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-cf49989db-x8vx4 1/1 Running 0 53s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-cf49989db-x8vx4 condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.20.19) [e2e-llm-inference-service] 🎯 Server version (4.20.19) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.19.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.19.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-bfc7f696d-wxdcx 1/1 Running 0 11s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-bfc7f696d-wxdcx condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-7cd77c7ffd-h4tbj 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-7cd77c7ffd-h4tbj condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-6b94bb86d8-j64f2 1/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2 condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.3.2 found, but not yet Succeeded (Phase: Installing). Waiting... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.3.2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.3.2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.3.2 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-7587b89b76-hnn8l 1/1 Running 0 67s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-7587b89b76-hnn8l condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-7587b89b76-hnn8l 1/1 Running 0 76s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-7587b89b76-hnn8l condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Now using project "kserve" on server "https://a692ccb1eeed14afa90aadf4cc55a129-e6f7d1a45d273665.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] ⏳ Installing KServe with SeaweedFS [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] ⏳ Waiting for CRDs to be established [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] ⏳ Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] ⏳ Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-56c5fd6bcc-x6vxb 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-56c5fd6bcc-x6vxb condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] ⏳ Re-applying LLMInferenceServiceConfig resources with webhook validation... [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-scheduler is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] Installing DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] Patching ingress domain, markers: llminferenceservice and cluster_cpu [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-6fc5d867c5-rx7vk" deleted [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc patched [e2e-llm-inference-service] waiting kserve-controller get ready... [e2e-llm-inference-service] pod/kserve-controller-manager-6fc5d867c5-mgfdv condition met [e2e-llm-inference-service] Installing ODH Model Controller manually with PR images [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] Warning: resource clusterroles/metrics-reader is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader configured [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init created [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.20.19 [e2e-llm-inference-service] Kubernetes Version: v1.33.9 [e2e-llm-inference-service] ClusterVersion desired: 4.20.19 [e2e-llm-inference-service] ClusterVersion history (latest): 4.20.19 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.3.0 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.3.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.3.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.3.2 Succeeded [e2e-llm-inference-service] servicemeshoperator3.v3.1.0 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.3.0 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.18.1-2 Succeeded [e2e-llm-inference-service] dns-operator.v1.3.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.3.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.3.2 Succeeded [e2e-llm-inference-service] servicemeshoperator3.v3.1.0 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.3.0 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.3.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.3.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.3.2 Succeeded [e2e-llm-inference-service] servicemeshoperator3.v3.1.0 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.3.0 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.3.0 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.3.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.3.2 Succeeded [e2e-llm-inference-service] servicemeshoperator3.v3.1.0 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.3.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.3.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.3.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.3.2 Succeeded [e2e-llm-inference-service] servicemeshoperator3.v3.1.0 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.3.0 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.3.0 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.3.0 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.3.2 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=-----BEGIN CERTIFICATE----- [e2e-llm-inference-service] MIIDPDCCAiSgAwIBAgIIWBjXmkJmTMgwDQYJKoZIhvcNAQELBQAwJjESMBAGA1UE [e2e-llm-inference-service] CxMJb3BlbnNoaWZ0MRAwDgYDVQQDEwdyb290LWNhMB4XDTI2MDQyMzE3MzQ0NloX [e2e-llm-inference-service] DTM2MDQyMDE3MzQ0NlowJjESMBAGA1UECxMJb3BlbnNoaWZ0MRAwDgYDVQQDEwdy [e2e-llm-inference-service] b290LWNhMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAujyRvQ7OATVc [e2e-llm-inference-service] trsxkhrs8FRxsCCdqgmFAS4e4t7in1YX/2wsei3Dsq9uOkMdeTwEXM5Cgmsz0/iS [e2e-llm-inference-service] g/Dman0MNJuK6UgQrVzVgRLSyK3k+7q5nqqR0NFEwH5ICHVcK/7JgC8sLf8BmDrv [e2e-llm-inference-service] KVxdb5moJkUJrGlX5Xf77iv4C9WWOEwiFHyaqTfZeerukOLCzceKxFkL4D0oW28I [e2e-llm-inference-service] 2e/+XEl+BPWP0CD1WmabeK1VRc3TjJEmEIjK5UmgyPvf+NX/fCEAOSl7ZLLmMIHv [e2e-llm-inference-service] mqgTpHVjNNsEKlYFKIrmXyOsCdn6W/rkBaJBbVQscHAqfU1X2snNb9JIPoJdY62O [e2e-llm-inference-service] R2a3Ut1qtwIDAQABo24wbDAOBgNVHQ8BAf8EBAMCAqQwDwYDVR0TAQH/BAUwAwEB [e2e-llm-inference-service] /zBJBgNVHQ4EQgRAEv/FVrDjGPa3p8vXbWwDiC1zT8rekO9pGOJUPOhW3VnKWs6q [e2e-llm-inference-service] GuuOlJ1dW60Gi6QZCjwyXKwicRpRA3E7Ek187DANBgkqhkiG9w0BAQsFAAOCAQEA [e2e-llm-inference-service] gcz/6Rcbw5nNGYLtHJeKDKNLN0SEgRuJBaQti4w0gHd8WNAH3L7Hiom2dJn92T8Y [e2e-llm-inference-service] 07n78WbQEubH+cy5dJMyHKippQ6pSFA07VF2eb/we5w6fAbjvmH4U25oqL/pGjNP [e2e-llm-inference-service] 7SQElJpNgP5utUGVN3lJGR9knwZBuMYhSUuY/XCEX/EkyQE5wh1t3gGOWkehMVtH [e2e-llm-inference-service] D995Rl0J+hDqBVI+KP68PtQ57TO5QL0n9Noq+AeBArkL1CVv1SB623vGfdC2C8ix [e2e-llm-inference-service] 0eaNz1/oW9e5DuvkZk9w7n0ldbEH8J2v7TNaRpFqKmyX/S4bXPQ1cR7bnJVrBCme [e2e-llm-inference-service] V6RSyCQ+PPIWUwP7SKOciA== [e2e-llm-inference-service] -----END CERTIFICATE----- [e2e-llm-inference-service] -----BEGIN CERTIFICATE----- [e2e-llm-inference-service] MIIEADCCAuigAwIBAgIIJ6FiXPc4dX0wDQYJKoZIhvcNAQELBQAwJjESMBAGA1UE [e2e-llm-inference-service] CxMJb3BlbnNoaWZ0MRAwDgYDVQQDEwdyb290LWNhMB4XDTI2MDQyMzE3MzUzM1oX [e2e-llm-inference-service] DTI3MDQyMzE3MzUzM1owMDESMBAGA1UEChMJb3BlbnNoaWZ0MRowGAYDVQQDExFv [e2e-llm-inference-service] cGVuc2hpZnQtaW5ncmVzczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB [e2e-llm-inference-service] AKQgV2xN6B6KhGeDPo+aMZx0IR7taQwixIE2YgHef+QuGT/hqi+sWSw7UOE2TE5b [e2e-llm-inference-service] 2O0Z5DrJyfNdKN4Q3GB3BrV7WcUVp1fGnczUABTC81LMHmNeN/3mthYGQQjsW7BY [e2e-llm-inference-service] LHX/oKrmboUmpN69OfTIoVLmsod6DmkLtDCXOENVllQhS5SI3a3z+ACI/a1fycJS [e2e-llm-inference-service] lt2KjmxoN3VWWcFKXwum5REXSMi3x+IH68pho484rRHSwy95sUguDqaoNsUj73K0 [e2e-llm-inference-service] o4Is8csr85+Y7enRYqreTtU5kT+N8kr3piEdik2Hylhy7Ft7MBe/gltk/Xbk8qfq [e2e-llm-inference-service] lY/7qAIKNaWYR6ByQSNEjKUCAwEAAaOCASYwggEiMA4GA1UdDwEB/wQEAwIFoDAd [e2e-llm-inference-service] BgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADBJBgNV [e2e-llm-inference-service] HQ4EQgRAaQmYmdaQnxmxf7UE/IBNe9nT0FKG3KsgeMLiKUDtw+mUWbM7jyoIBCkS [e2e-llm-inference-service] 0MtBds39D8LA9g+nlhe6udlIgTFf1zBLBgNVHSMERDBCgEAS/8VWsOMY9reny9dt [e2e-llm-inference-service] bAOILXNPyt6Q72kY4lQ86FbdWcpazqoa646UnV1brQaLpBkKPDJcrCJxGlEDcTsS [e2e-llm-inference-service] TXzsMEsGA1UdEQREMEKCQCouYXBwcy4xMjE4MzdkMi1mNGZiLTQ4MDUtYjE4MS00 [e2e-llm-inference-service] Mjk0NzVlNTRlMmYucHJvZC5rb25mbHV4ZWFhcy5jb20wDQYJKoZIhvcNAQELBQAD [e2e-llm-inference-service] ggEBABe2lN7es3hlCJeF3QqKlq2Hax2sy0HLSsCCwNhNPj3T9FykBRNdFQn5fkR4 [e2e-llm-inference-service] q1GMLXSRHfYBGo3R72gM2LH2Xd15PhPa38Zf0waEqboQCBYsDj8r120tDzGHEOVo [e2e-llm-inference-service] 5GplavE0gDQYBEOW88zOhpLFFPx5dLsd+OqA05j9XlyiXCan6fLJeNmeyaxhwfsf [e2e-llm-inference-service] HGBRuIN7byTqcsbxHcqg+Eca2peSJnjsItsV8vlX1Pyzx7oZc1lby9UhE02e8WA1 [e2e-llm-inference-service] oPK8kedm81+ETX2F/0Fx+xJAQLOWrs4aq2MGipOqw3gqZiAO1uJrf+HxeK4BQnNJ [e2e-llm-inference-service] cmXNlrk30/bSE8q7zaKlp8C1EAg= [e2e-llm-inference-service] -----END CERTIFICATE----- [e2e-llm-inference-service] [e2e-llm-inference-service] -----BEGIN CERTIFICATE----- [e2e-llm-inference-service] MIIDUTCCAjmgAwIBAgIISzNG/UpxGT0wDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE [e2e-llm-inference-service] Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc3Njk2NjIzMTAe [e2e-llm-inference-service] Fw0yNjA0MjMxNzQzNTBaFw0yODA2MjExNzQzNTFaMDYxNDAyBgNVBAMMK29wZW5z [e2e-llm-inference-service] aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3NzY5NjYyMzEwggEiMA0GCSqG [e2e-llm-inference-service] SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDY9XHXoj7QW8VXl+O2DIfmTk7xY3Ivx0IT [e2e-llm-inference-service] qa1W7AKk0Y9LYJC12cCT2L+wkEet9pHUiP24WH/UoSwZ3yE4ayRg4FO3ILHGPAWc [e2e-llm-inference-service] RViSozgKFEVwm3WfFVRGv4lf7zKBXBSjL/BIF3XPxJK8fSeQUitdtUOYGRG6yR8A [e2e-llm-inference-service] 719N4gILZFWT7pTcS0aiTBJ0JHhKmHCXaTg9ipT/pkrtGRBVp1HHWHPuSiR5xF6i [e2e-llm-inference-service] MVvq8s6yArUkCCiohlrqTUscVaZV+wC8pTdbWfYBosK2Xa1jezYllp9R2xrWDQi/ [e2e-llm-inference-service] vVXaoUvFW/bJvuXW/YwO2JrPsU5okituv/A3rJmGGYAVDQWkp2PfAgMBAAGjYzBh [e2e-llm-inference-service] MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBT7xIuy [e2e-llm-inference-service] 1uZdgKsXnfB/x4nrSvf+TDAfBgNVHSMEGDAWgBT7xIuy1uZdgKsXnfB/x4nrSvf+ [e2e-llm-inference-service] TDANBgkqhkiG9w0BAQsFAAOCAQEAmS0XgeXEHFf4owq1AxhTKDAhvqEUk9w0Sm2J [e2e-llm-inference-service] o8PehHEALLqMdfrPXkFVIf+t2fZQ073pn1Sr3HXHicb9HGSV0Fy/kA9K8mrFKejH [e2e-llm-inference-service] fzm+gSD8/PaHod6plSniRQiF7pag+qtzQePiNDTd57/EpbC1e8jn5ugb/8dhLllL [e2e-llm-inference-service] o/7O7VVMeMpTzT6oybjwUtpVGOD6u2+X67sRyy4Qk/Cb6lsr/H4rFYAnYV/eSELz [e2e-llm-inference-service] JMVt5xxUd41XFLofzCqM6MDTpcT4KzXJ02XuTdzsawTQMKfL1zO9nRrf4ksoXlE6 [e2e-llm-inference-service] nnrdkCkYIeMjeN4hwoE5bc7gZOaO9v9i3RVW1OjrB33WfsIehA== [e2e-llm-inference-service] -----END CERTIFICATE----- [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: anyio-4.9.0, cov-5.0.0, asyncio-0.23.8, xdist-3.6.1, httpx-0.30.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [22 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] 2026-04-23 18:21:31.713 5272 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-04-23 18:21:31.713 5275 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-04-23 18:21:31.725 5272 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 18:21:31.725 5272 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 18:21:31.747 5272 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-04-23 18:21:31.747 5272 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-04-23 18:21:31.754 5272 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-04-23 18:21:31.754 5272 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-04-23 18:21:54.497 5272 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 18:21:54.497 5272 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 18:21:54.540 5272 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-04-23 18:21:54.540 5272 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] 2026-04-23 18:22:30.857 5272 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 18:22:30.857 5272 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 18:22:30.902 5272 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-04-23 18:22:30.902 5272 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] 2026-04-23 19:03:32.783 5272 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 19:03:32.783 5272 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 19:03:32.832 5272 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-04-23 19:03:32.832 5272 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-04-23 19:03:32.832 5272 kserve.trace Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 19:03:32.832 5272 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 19:03:32.840 5272 kserve.trace Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-04-23 19:03:32.840 5272 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-04-23 19:03:32.854 5272 kserve.trace ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-04-23 19:03:32.854 5272 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-04-23 19:03:32.854 5272 kserve.trace Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 19:03:32.854 5272 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 19:03:32.858 5272 kserve.trace Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-04-23 19:03:32.858 5272 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-04-23 19:03:32.870 5272 kserve.trace ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] 2026-04-23 19:03:32.870 5272 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] 2026-04-23 19:13:24.333 5272 kserve.trace Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 19:13:24.333 5272 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 19:13:24.366 5272 kserve.trace Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-04-23 19:13:24.366 5272 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-04-23 19:13:24.376 5272 kserve.trace ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-04-23 19:13:24.376 5272 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-04-23 19:13:24.377 5272 kserve.trace Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 19:13:24.377 5272 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 19:13:24.380 5272 kserve.trace Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-04-23 19:13:24.380 5272 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-04-23 19:13:24.393 5272 kserve.trace ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-04-23 19:13:24.393 5272 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-04-23 19:13:24.393 5272 kserve.trace Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 19:13:24.393 5272 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-04-23 19:13:24.400 5272 kserve.trace Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-04-23 19:13:24.400 5272 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-04-23 19:13:24.412 5272 kserve.trace ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] 2026-04-23 19:13:24.412 5272 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] _ test_llm_inference_service[router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-with-gateway-ref', 'router-with-managed-route', 'model-fb-opt-125m', 'workload-llmd-simula... {'name': 'workload-llmd-simulator-llmisvc-e25babec'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:410: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...model-52a2b82b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-e25babec'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-04-23T18:22:32.073685', start_time = 1776968552.074157 [e2e-llm-inference-service] duration = 900.572124004364, timestamp_end = '2026-04-23T18:37:32.646281' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....misvc-model-52a2b82b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-e25babec'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:618: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7fb9c4b8d080> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:628: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:613: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1107 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:34 Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:57 ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1030 ✓ Created/updated Gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1056 Checking LLMInferenceServiceConfig router-with-gateway-ref-llmisvc-8a6d75cb in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1082 Resource not found, creating LLMInferenceServiceConfig router-with-gateway-ref-llmisvc-8a6d75cb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1092 ✓ Successfully created LLMInferenceServiceConfig router-with-gateway-ref-llmisvc-8a6d75cb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1056 Checking LLMInferenceServiceConfig router-with-managed-route-llmis-615ab0ab in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1082 Resource not found, creating LLMInferenceServiceConfig router-with-managed-route-llmis-615ab0ab [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1092 ✓ Successfully created LLMInferenceServiceConfig router-with-managed-route-llmis-615ab0ab [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1056 Checking LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-52a2b82b in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1082 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-52a2b82b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1092 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-52a2b82b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1056 Checking LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-e25babec in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1082 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-e25babec [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1092 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-e25babec [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-04-23T18:22:31.885865] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-with-gateway-ref', 'router-with-managed-route', 'model-fb-opt-125m', 'workload-llmd-simulator'], prompt='KServe is a', service_name='llmisvc-model-fb-opt-125m-route-6fa1027a', endpoint='/v1/completions', max_tokens=100, payload_formatter=, response_assertion=.response_assertion at 0x7fb9c514c400>, wait_timeout=900, response_timeout=60, before_test=[ at 0x7fb9c514c4a0>], after_test=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-6fa1027a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-with-gateway-ref-llmisvc-8a6d75cb'}, [e2e-llm-inference-service] {'name': 'router-with-managed-route-llmis-615ab0ab'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-52a2b82b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-e25babec'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1107 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-04-23T18:22:31.898496] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-6fa1027a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-with-gateway-ref-llmisvc-8a6d75cb'}, [e2e-llm-inference-service] {'name': 'router-with-managed-route-llmis-615ab0ab'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-52a2b82b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-e25babec'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-04-23T18:22:32.073461] end - ✅ in 0.175s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-04-23T18:22:32.073685] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-6fa1027a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-with-gateway-ref-llmisvc-8a6d75cb'}, [e2e-llm-inference-service] {'name': 'router-with-managed-route-llmis-615ab0ab'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-52a2b82b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-e25babec'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:32.074165] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:32.080383] end - ✅ in 0.006s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:33.080657] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:33.173325] end - ✅ in 0.092s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:34.173653] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:34.181242] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:35.181501] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:35.188732] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:36.189109] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:36.273572] end - ✅ in 0.084s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:37.273950] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:37.280826] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:38.281138] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:38.288761] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:39.289212] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:39.373511] end - ✅ in 0.084s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:40.373819] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:40.475625] end - ✅ in 0.102s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:41.475983] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:41.494474] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:42.495095] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:42.573727] end - ✅ in 0.078s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:43.574123] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:43.582129] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:44.582414] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:44.590245] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:45.590568] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:45.598524] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:46.598809] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:46.607020] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:47.607312] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:47.615093] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:48.615369] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:48.623300] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:49.623716] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:49.632627] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:50.632878] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:50.640578] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:51.640924] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:51.649485] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:52.649857] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:52.660828] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:53.661241] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:53.669568] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:54.669916] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:54.678357] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:55.678689] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:55.686283] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:56.686567] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:56.695540] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:57.695795] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:57.704475] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:58.704816] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:58.713367] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:59.713652] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:59.721736] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:00.722052] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:00.730065] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:01.730323] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:01.741739] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:02.742175] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:02.750576] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:03.750954] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:03.758554] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:04.758849] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:04.767194] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:05.767488] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:05.779517] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:06.779870] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:06.788495] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:07.788776] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:07.797962] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:08.798450] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:08.807024] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:09.807364] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:09.815376] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:10.815673] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:10.823164] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:11.823418] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:11.831005] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:12.831283] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:12.839288] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:13.839617] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:13.849829] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:14.850142] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:14.858415] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:15.858705] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:15.866827] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:16.867152] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:16.875828] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:17.876225] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:17.884443] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:18.884739] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:18.893478] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:19.893779] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:19.901675] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:20.901975] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:20.910142] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:21.910588] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:21.919546] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:22.919961] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:22.928560] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:23.929039] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:23.937267] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:24.937638] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:24.945468] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:25.945844] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:25.953583] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:26.953881] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:26.961872] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:27.962168] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:27.970165] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:28.970432] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:28.978785] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:29.979220] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:29.987417] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:30.987697] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:30.995798] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:31.996160] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:32.005235] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:33.005512] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:33.013851] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:34.014277] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:34.022132] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:35.022411] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:35.030097] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:36.030387] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:36.038224] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:37.038478] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:37.047473] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:38.047798] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:38.056112] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:39.056427] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:39.064576] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:40.064854] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:40.072983] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:41.073296] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:41.082234] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:42.082526] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:42.090294] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:43.090679] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:43.099539] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:44.099940] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:44.108544] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:45.108841] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:45.118641] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:46.118946] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:46.127327] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:47.127766] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:47.136515] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:48.136863] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:48.145279] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:49.145661] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:49.153704] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:50.154054] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:50.162836] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:51.163149] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:51.171894] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:52.172212] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:52.181687] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:53.182056] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:53.190754] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:54.191111] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:54.199255] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:55.199609] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:55.207859] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:56.208174] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:56.216220] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:57.216572] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:57.225078] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:58.225372] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:58.233502] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:23:59.233887] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:23:59.241822] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:00.242200] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:00.250946] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:01.251293] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:01.260202] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:02.260477] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:02.268391] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:03.268710] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:03.276308] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:04.276612] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:04.284429] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:05.284713] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:05.293234] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:06.293530] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:06.302226] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:07.302473] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:07.310643] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:08.310954] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:08.319204] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:09.319468] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:09.327348] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:10.327608] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:10.335379] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:11.335675] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:11.344410] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:12.344684] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:12.354847] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:13.355371] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:13.362668] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:14.362952] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:14.371262] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:15.371539] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:15.379315] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:16.379607] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:16.387965] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:17.388355] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:17.396568] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:18.396828] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:18.405466] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:19.405882] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:19.414375] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:20.414844] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:20.422659] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:21.422989] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:21.431504] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:22.431952] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:22.440234] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:23.440506] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:23.448402] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:24.448737] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:24.457594] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:25.458050] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:25.467193] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:26.467533] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:26.477095] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:27.477382] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:27.485463] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:28.485926] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:28.494459] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:29.494738] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:29.503258] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:30.503636] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:30.511882] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:31.512255] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:31.519837] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:32.520182] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:32.529580] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:33.529851] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:33.537988] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:34.538371] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:34.546063] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:35.546355] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:35.555106] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:36.555441] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:36.563360] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:37.563665] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:37.571617] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:38.571924] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:38.580451] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:39.580693] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:39.588251] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:40.588537] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:40.596659] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:41.596949] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:41.604521] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:42.604807] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:42.612461] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:43.612761] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:43.620673] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:44.620968] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:44.628392] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:45.628878] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:45.638062] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:46.638361] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:46.646143] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:47.646427] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:47.654304] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:48.654603] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:48.662737] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:49.663064] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:49.670858] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:50.671138] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:50.679995] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:51.680352] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:51.688496] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:52.688932] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:52.697548] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:53.697932] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:53.705942] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:54.706214] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:54.714367] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:55.714669] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:55.723225] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:56.723531] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:56.731649] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:57.732123] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:57.743129] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:58.743453] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:58.751833] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:24:59.752268] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:24:59.761132] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:00.761381] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:00.770211] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:01.770469] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:01.778471] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:02.778749] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:02.786878] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:03.787321] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:03.796058] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:04.796368] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:04.804419] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:05.804677] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:05.813405] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:06.813715] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:06.821718] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:07.822261] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:07.830521] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:08.830765] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:08.840103] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:09.840393] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:09.848271] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:10.848558] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:10.856799] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:11.857159] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:11.867110] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:12.867403] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:12.875593] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:13.875870] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:13.884101] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:14.884402] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:14.892407] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:15.892710] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:15.901218] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:16.901515] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:16.909679] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:17.910053] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:17.918601] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:18.918948] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:18.932789] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:19.933090] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:19.941494] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:20.941818] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:20.952618] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:21.952950] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:21.962361] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:22.962667] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:22.971059] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:23.971358] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:23.979522] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:24.979844] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:24.988125] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:25.988390] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:25.997551] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:26.998074] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:27.006225] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:28.006531] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:28.014483] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:29.014790] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:29.026080] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:30.026391] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:30.034519] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:31.034802] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:31.042284] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:32.042628] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:32.050472] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:33.050728] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:33.058522] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:34.058796] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:34.072111] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:35.072525] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:35.081291] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:36.081553] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:36.089683] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:37.089984] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:37.098677] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:38.098950] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:38.108256] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:39.108649] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:39.121422] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:40.121709] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:40.130172] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:41.130512] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:41.139855] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:42.140135] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:42.148315] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:43.148643] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:43.157750] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:44.157993] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:44.166311] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:45.166637] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:45.175590] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:46.175955] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:46.188758] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:47.189039] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:47.197487] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:48.197791] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:48.206102] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:49.206432] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:49.214325] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:50.214608] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:50.222964] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:51.223257] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:51.231557] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:52.231843] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:52.240406] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:53.240721] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:53.248684] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:54.249002] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:54.257733] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:55.258166] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:55.266865] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:56.267302] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:56.275461] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:57.275775] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:57.283658] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:58.283957] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:58.292172] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:25:59.292491] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:25:59.300405] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:00.300714] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:00.309748] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:01.310114] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:01.318692] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:02.318960] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:02.326993] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:03.327281] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:03.335489] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:04.335828] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:04.343743] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:05.344098] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:05.351949] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:06.352300] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:06.360136] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:07.360395] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:07.368638] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:08.368956] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:08.377207] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:09.377501] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:09.385318] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:10.385610] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:10.393886] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:11.394268] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:11.403018] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:12.403348] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:12.411370] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:13.411735] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:13.419766] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:14.420107] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:14.428634] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:15.428894] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:15.437392] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:16.437777] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:16.446491] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:17.446853] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:17.455595] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:18.455955] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:18.464142] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:19.464449] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:19.473700] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:20.474109] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:20.482282] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:21.482588] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:21.490763] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:22.491182] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:22.499159] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:23.499433] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:23.507380] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:24.507673] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:24.515552] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:25.515833] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:25.523541] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:26.523827] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:26.531922] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:27.532266] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:27.553335] end - ✅ in 0.021s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:28.553610] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:28.561186] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:29.561457] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:29.569354] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:30.569714] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:30.579480] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:31.579729] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:31.587838] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:32.588175] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:32.595976] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:33.596271] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:33.603969] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:34.604293] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:34.612183] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:35.612487] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:35.623195] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:36.623541] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:36.631001] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:37.631302] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:37.639339] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:38.639722] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:38.647561] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:39.647856] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:39.656297] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:40.656704] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:40.737244] end - ✅ in 0.080s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:41.737581] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:41.746068] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:42.746423] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:42.766037] end - ✅ in 0.019s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:43.766350] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:43.774288] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:44.774595] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:44.782498] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:45.782786] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:45.790868] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:46.791151] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:46.798879] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:47.799151] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:47.807327] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:48.807625] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:48.816176] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:49.816447] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:49.825025] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:50.825308] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:50.833708] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:51.834106] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:51.842699] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:52.843143] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:52.851880] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:53.852209] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:53.860082] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:54.860335] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:54.867975] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:55.868241] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:55.876374] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:56.876673] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:56.884202] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:57.884475] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:57.893281] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:58.893555] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:58.902239] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:26:59.902576] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:26:59.910767] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:00.911247] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:00.919384] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:01.919666] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:01.928062] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:02.928445] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:02.936460] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:03.936710] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:03.944978] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:04.945432] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:04.953782] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:05.954290] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:05.962593] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:06.962937] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:06.971490] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:07.971786] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:07.980593] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:08.980870] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:08.992658] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:09.992930] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:10.001400] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:11.001759] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:11.010479] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:12.010827] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:12.019479] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:13.019826] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:13.030358] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:14.030680] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:14.040163] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:15.040487] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:15.048585] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:16.048890] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:16.057190] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:17.057473] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:17.065498] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:18.065871] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:18.074783] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:19.075227] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:19.083857] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:20.084192] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:20.092432] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:21.092756] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:21.101609] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:22.101890] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:22.110813] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:23.111099] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:23.119249] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:24.119578] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:24.127533] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:25.127861] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:25.136161] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:26.136547] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:26.145500] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:27.145829] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:27.153984] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:28.154290] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:28.162132] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:29.162390] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:29.170815] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:30.171319] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:30.179658] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:31.179947] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:31.188152] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:32.188542] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:32.197438] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:33.197720] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:33.205523] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:34.205790] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:34.213869] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:35.214226] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:35.222281] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:36.222563] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:36.230228] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:37.230542] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:37.239222] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:38.239717] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:38.247533] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:39.247815] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:39.256733] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:40.257007] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:40.265232] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:41.265519] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:41.273544] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:42.273854] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:42.282267] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:43.282621] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:43.291047] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:44.291357] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:44.301456] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:45.301724] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:45.310563] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:46.310983] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:46.327766] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:47.328109] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:47.337575] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:48.338009] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:48.346292] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:49.346624] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:49.354666] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:50.354962] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:50.363094] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:51.363375] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:51.371712] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:52.372183] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:52.380150] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:53.380428] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:53.388220] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:54.388510] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:54.396314] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:55.396616] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:55.405532] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:56.405804] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:56.415094] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:57.415415] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:57.425882] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:58.426426] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:58.434467] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:27:59.434717] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:27:59.442457] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:00.442702] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:00.450569] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:01.450866] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:01.459412] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:02.459660] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:02.467731] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:03.468098] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:03.476495] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:04.476771] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:04.485071] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:05.485365] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:05.494054] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:06.494485] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:06.503613] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:07.503958] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:07.512173] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:08.512486] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:08.520814] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:09.521119] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:09.528776] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:10.529139] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:10.537519] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:11.537822] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:11.545985] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:12.546249] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:12.560897] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:13.561215] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:13.569654] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:14.569934] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:14.578303] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:15.578586] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:15.586339] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:16.586624] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:16.594623] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:17.594963] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:17.602839] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:18.603136] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:18.611142] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:19.611459] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:19.620092] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:20.620377] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:20.631131] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:21.631379] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:21.639884] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:22.640201] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:22.648338] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:23.648610] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:23.656473] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:24.656787] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:24.665306] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:25.665571] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:25.673685] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:26.673955] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:26.681886] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:27.682212] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:27.690326] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:28.690621] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:28.698605] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:29.698876] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:29.707335] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:30.707639] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:30.716428] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:31.716726] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:31.725301] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:32.725609] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:32.733199] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:33.733481] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:33.741402] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:34.741683] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:34.750175] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:35.750466] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:35.758420] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:36.758748] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:36.768323] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:37.768722] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:37.776753] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:38.777112] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:38.785240] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:39.785664] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:39.793413] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:40.793702] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:40.801787] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:41.802067] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:41.810708] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:42.811058] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:42.819384] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:43.819697] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:43.827655] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:44.827924] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:44.835964] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:45.836223] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:45.844447] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:46.844889] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:46.853454] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:47.853728] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:47.863357] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:48.863664] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:48.871385] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:49.871684] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:49.879414] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:50.879717] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:50.887169] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:51.887447] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:51.896168] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:52.896447] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:52.904681] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:53.905044] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:53.913271] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:54.913528] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:54.921408] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:55.921700] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:55.930340] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:56.930615] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:56.938932] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:57.939347] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:57.947989] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:58.948413] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:58.957531] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:59.957853] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:59.966758] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:00.967073] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:00.975483] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:01.975732] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:01.983702] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:02.984037] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:02.992182] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:03.992510] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:04.000528] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:05.000803] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:05.009635] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:06.009959] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:06.018428] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:07.018789] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:07.026806] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:08.027202] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:08.035539] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:09.035890] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:09.044670] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:10.044994] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:10.053108] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:11.053420] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:11.061422] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:12.061803] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:12.070566] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:13.070956] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:13.080650] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:14.081065] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:14.089319] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:15.089600] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:15.097534] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:16.097843] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:16.106880] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:17.107224] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:17.115730] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:18.116189] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:18.124153] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:19.124426] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:19.132332] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:20.132636] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:20.140466] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:21.140775] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:21.149808] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:22.150276] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:22.159576] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:23.159881] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:23.167645] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:24.167921] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:24.176304] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:25.176565] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:25.184860] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:26.185293] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:26.193436] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:27.193743] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:27.201480] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:28.201760] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:28.209448] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:29.209722] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:29.219283] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:30.219629] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:30.227418] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:31.227705] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:31.235811] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:32.236265] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:32.245101] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:33.245383] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:33.253031] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:34.253351] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:34.261710] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:35.262028] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:35.270352] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:36.270640] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:36.279568] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:37.279854] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:37.287764] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:38.288038] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:38.295357] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:39.295660] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:39.303986] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:40.304273] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:40.311653] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:41.311934] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:41.320511] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:42.320829] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:42.328973] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:43.329229] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:43.337085] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:44.337379] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:44.353713] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:45.354111] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:45.365372] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:46.365719] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:46.374000] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:47.374267] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:47.382291] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:48.382547] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:48.390448] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:49.390780] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:49.398566] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:50.398881] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:50.407319] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:51.407663] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:51.416648] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:52.417150] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:52.425161] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:53.425530] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:53.433348] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:54.433620] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:54.441527] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:55.441820] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:55.450612] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:56.450892] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:56.458688] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:57.458969] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:57.467368] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:58.467650] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:58.475639] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:29:59.475954] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:29:59.489076] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:00.489544] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:00.498483] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:01.498861] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:01.507090] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:02.507466] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:02.515988] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:03.516309] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:03.524933] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:04.525420] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:04.533643] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:05.534145] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:05.542487] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:06.542771] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:06.551379] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:07.551722] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:07.559515] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:08.559873] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:08.568850] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:09.569126] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:09.577783] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:10.578065] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:10.586061] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:11.586337] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:11.594549] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:12.594851] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:12.604014] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:13.604406] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:13.612474] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:14.612853] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:14.621197] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:15.621654] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:15.629659] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:16.630096] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:16.638203] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:17.638618] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:17.646850] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:18.647201] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:18.655840] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:19.656211] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:19.664507] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:20.664828] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:20.673389] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:21.673647] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:21.681999] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:22.682245] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:22.698975] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:23.699429] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:23.707532] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:24.707863] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:24.716249] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:25.716572] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:25.724572] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:26.724890] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:26.732532] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:27.732843] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:27.743857] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:28.744266] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:28.752419] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:29.752686] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:29.760729] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:30.761060] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:30.768610] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:31.768957] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:31.777727] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:32.778028] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:32.786211] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:33.786525] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:33.794615] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:34.794952] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:34.805241] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:35.805586] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:35.813720] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:36.813989] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:36.833605] end - ✅ in 0.019s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:37.833970] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:37.843663] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:38.844037] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:38.852625] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:39.852946] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:39.861212] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:40.861593] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:40.870192] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:41.870498] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:41.878663] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:42.879095] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:42.887256] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:43.887558] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:43.895712] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:44.896304] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:44.904592] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:45.905088] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:45.914104] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:46.914351] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:46.926041] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:47.926358] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:47.934651] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:48.935153] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:48.944484] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:49.944965] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:49.953290] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:50.953574] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:50.962140] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:51.962458] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:51.971536] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:52.971823] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:52.979927] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:53.980238] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:53.988577] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:54.988822] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:54.996540] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:55.996824] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:56.007746] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:57.008097] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:57.017062] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:58.017518] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:58.026472] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:30:59.026751] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:30:59.035834] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:00.036287] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:00.044539] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:01.044814] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:01.053430] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:02.053769] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:02.062525] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:03.062854] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:03.071342] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:04.071774] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:04.080739] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:05.081026] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:05.094543] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:06.094946] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:06.112769] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:07.113105] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:07.121832] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:08.122118] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:08.129501] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:09.129787] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:09.137600] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:10.137880] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:10.146501] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:11.146776] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:11.155333] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:12.155640] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:12.164545] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:13.164966] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:13.173028] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:14.173302] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:14.182222] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:15.182507] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:15.192225] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:16.192501] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:16.201320] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:17.201650] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:17.213827] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:18.214273] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:18.222526] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:19.222848] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:19.231375] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:20.231660] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:20.241129] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:21.241464] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:21.250847] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:22.251367] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:22.260393] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:23.260671] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:23.269373] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:24.269736] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:24.279627] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:25.279977] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:25.287607] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:26.287859] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:26.295884] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:27.296226] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:27.304506] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:28.304838] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:28.313171] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:29.313621] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:29.322215] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:30.322671] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:30.331511] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:31.331837] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:31.340701] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:32.340973] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:32.348977] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:33.349373] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:33.357548] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:34.357802] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:34.366041] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:35.366364] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:35.374611] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:36.374982] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:36.383533] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:37.383835] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:37.391956] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:38.392239] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:38.399886] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:39.400166] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:39.407688] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:40.407981] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:40.415421] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:41.415728] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:41.423856] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:42.424210] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:42.431964] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:43.432275] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:43.440613] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:44.441110] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:44.449533] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:45.449952] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:45.457792] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:46.458172] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:46.466463] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:47.466718] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:47.474404] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:48.474687] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:48.482594] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:49.482956] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:49.491892] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:50.492344] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:50.503875] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:51.504476] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:51.513341] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:52.513623] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:52.521948] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:53.522421] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:53.530242] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:54.530509] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:54.538234] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:55.538547] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:55.547467] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:56.547762] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:56.556204] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:57.556637] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:57.565174] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:58.565434] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:58.573552] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:31:59.573836] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:31:59.582342] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:00.582645] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:00.591503] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:01.591843] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:01.600459] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:02.600817] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:02.609200] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:03.609764] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:03.618683] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:04.618956] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:04.629606] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:05.629966] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:05.639126] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:06.639566] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:06.647633] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:07.647954] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:07.655882] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:08.656214] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:08.664616] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:09.665079] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:09.673816] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:10.674175] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:10.682741] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:11.683130] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:11.691755] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:12.692196] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:12.700599] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:13.701004] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:13.709007] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:14.709312] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:14.717659] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:15.717949] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:15.726497] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:16.726798] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:16.734879] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:17.735299] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:17.745438] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:18.745759] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:18.754084] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:19.754360] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:19.762176] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:20.762488] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:20.770556] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:21.770882] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:21.780690] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:22.780977] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:22.788758] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:23.789239] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:23.797389] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:24.797661] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:24.805757] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:25.806065] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:25.813886] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:26.814241] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:26.822387] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:27.822675] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:27.833559] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:28.833853] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:28.842593] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:29.842939] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:29.851606] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:30.851854] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:30.859666] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:31.859935] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:31.868383] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:32.868844] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:32.878223] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:33.878548] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:33.886149] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:34.886413] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:34.894523] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:35.894863] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:35.902540] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:36.902819] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:36.911041] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:37.911336] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:37.919250] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:38.919545] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:38.928749] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:39.929149] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:39.937466] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:40.937887] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:40.946076] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:41.946355] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:41.954881] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:42.955331] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:42.963618] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:43.963944] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:43.972368] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:44.972650] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:44.980874] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:45.981340] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:45.990047] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:46.990420] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:47.000437] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:48.000744] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:48.011199] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:49.011474] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:49.019958] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:50.020365] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:50.028955] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:51.029419] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:51.037447] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:52.037749] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:52.046466] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:53.046720] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:53.055246] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:54.055555] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:54.064177] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:55.064557] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:55.072612] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:56.072944] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:56.080558] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:57.080821] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:57.089143] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:58.089614] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:58.099356] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:32:59.099712] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:32:59.108429] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:00.108711] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:00.117314] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:01.117594] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:01.126481] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:02.126753] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:02.134787] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:03.135068] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:03.143086] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:04.143398] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:04.151667] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:05.151987] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:05.160458] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:06.160925] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:06.169129] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:07.169489] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:07.177927] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:08.178211] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:08.186653] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:09.187080] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:09.195440] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:10.195853] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:10.203445] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:11.203735] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:11.211571] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:12.211981] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:12.219699] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:13.220010] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:13.227712] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:14.228218] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:14.236187] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:15.236459] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:15.244566] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:16.244977] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:16.253472] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:17.253833] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:17.261603] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:18.261897] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:18.269925] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:19.270248] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:19.278276] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:20.278556] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:20.287064] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:21.287372] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:21.296353] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:22.296702] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:22.305174] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:23.305578] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:23.313438] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:24.313727] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:24.322214] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:25.322584] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:25.330809] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:26.331279] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:26.339849] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:27.340142] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:27.348218] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:28.348528] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:28.356778] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:29.357210] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:29.365244] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:30.365549] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:30.374238] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:31.374651] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:31.382113] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:32.382426] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:32.393213] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:33.393665] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:33.402378] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:34.402804] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:34.410951] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:35.411260] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:35.419160] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:36.419499] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:36.427389] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:37.427704] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:37.437580] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:38.437945] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:38.446637] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:39.447020] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:39.455573] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:40.455976] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:40.463518] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:41.463828] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:41.471230] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:42.471532] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:42.478891] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:43.479222] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:43.487227] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:44.487502] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:44.496461] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:45.496731] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:45.504380] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:46.504722] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:46.512343] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:47.512630] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:47.520351] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:48.520661] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:48.528714] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:49.529062] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:49.537045] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:50.537358] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:50.545950] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:51.546235] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:51.555037] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:52.555318] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:52.563290] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:53.563578] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:53.571670] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:54.571948] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:54.583511] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:55.583807] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:55.591718] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:56.591954] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:56.599790] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:57.600133] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:57.608561] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:58.608856] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:58.617480] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:33:59.617757] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:33:59.625652] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:00.625973] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:00.633830] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:01.634239] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:01.642760] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:02.643050] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:02.652686] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:03.652984] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:03.661446] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:04.661892] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:04.670518] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:05.670943] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:05.678872] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:06.679183] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:06.687785] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:07.688058] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:07.695965] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:08.696267] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:08.704317] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:09.704740] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:09.713369] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:10.713787] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:10.722856] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:11.723164] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:11.731707] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:12.732176] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:12.740843] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:13.741252] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:13.749849] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:14.750260] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:14.758713] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:15.759197] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:15.767087] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:16.767379] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:16.776040] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:17.776351] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:17.784997] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:18.785363] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:18.803727] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:19.804088] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:19.812027] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:20.812424] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:20.821390] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:21.821666] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:21.829522] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:22.829813] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:22.837658] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:23.838195] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:23.846321] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:24.846628] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:24.855857] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:25.856345] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:25.864628] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:26.865013] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:26.873332] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:27.873679] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:27.882406] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:28.882748] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:28.891611] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:29.891894] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:29.905260] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:30.905718] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:30.915202] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:31.915675] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:31.924233] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:32.924739] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:32.933520] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:33.933897] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:33.942946] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:34.943234] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:34.952568] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:35.952980] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:35.961456] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:36.961935] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:36.970311] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:37.970623] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:37.979046] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:38.979336] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:39.006040] end - ✅ in 0.026s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:40.006403] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:40.014799] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:41.015212] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:41.026002] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:42.026284] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:42.034879] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:43.035262] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:43.043822] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:44.044253] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:44.052797] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:45.053255] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:45.061824] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:46.062273] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:46.071005] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:47.071364] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:47.080030] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:48.080358] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:48.088356] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:49.088646] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:49.097145] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:50.097402] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:50.105089] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:51.105441] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:51.114008] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:52.114441] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:52.122880] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:53.123219] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:53.131682] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:54.132139] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:54.141244] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:55.141582] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:55.150493] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:56.150795] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:56.169639] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:57.169960] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:57.177675] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:58.177956] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:58.185660] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:59.186212] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:59.194726] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:00.194999] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:00.203167] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:01.203502] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:01.212001] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:02.212411] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:02.221073] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:03.221491] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:03.228776] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:04.229095] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:04.237392] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:05.237864] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:05.246142] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:06.246396] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:06.257373] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:07.257686] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:07.266118] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:08.266414] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:08.275565] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:09.276022] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:09.284969] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:10.285234] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:10.293402] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:11.293815] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:11.302137] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:12.302408] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:12.310353] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:13.310716] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:13.325051] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:14.325318] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:14.333183] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:15.333524] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:15.341808] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:16.342305] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:16.351170] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:17.351456] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:17.359399] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:18.359711] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:18.368492] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:19.368813] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:19.377629] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:20.378252] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:20.385733] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:21.386251] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:21.396422] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:22.396666] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:22.405016] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:23.405464] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:23.421347] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:24.421629] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:24.429733] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:25.430266] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:25.438517] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:26.438800] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:26.447532] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:27.447817] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:27.456509] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:28.456798] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:28.466082] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:29.466532] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:29.475637] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:30.475937] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:30.484745] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:31.485025] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:31.493484] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:32.493872] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:32.502428] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:33.502860] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:33.511968] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:34.512272] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:34.520599] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:35.520964] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:35.529712] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:36.530085] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:36.538492] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:37.538970] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:37.547575] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:38.548025] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:38.556850] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:39.557194] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:39.565807] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:40.566196] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:40.575549] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:41.576017] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:41.584477] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:42.584787] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:42.592950] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:43.593315] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:43.601689] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:44.602184] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:44.609613] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:45.609969] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:45.618307] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:46.618585] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:46.627299] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:47.627542] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:47.635619] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:48.636120] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:48.644761] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:49.645302] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:49.654320] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:50.654572] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:50.662777] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:51.663134] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:51.671837] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:52.672321] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:52.680375] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:53.680691] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:53.689271] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:54.689600] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:54.698077] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:55.698528] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:55.706828] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:56.707351] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:56.716233] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:57.716640] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:57.724722] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:58.725073] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:58.733597] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:35:59.733968] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:35:59.742202] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:00.742648] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:00.751712] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:01.751988] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:01.760252] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:02.760684] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:02.768714] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:03.769033] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:03.777150] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:04.777456] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:04.785478] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:05.785950] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:05.794053] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:06.794348] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:06.802736] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:07.803072] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:07.811469] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:08.811798] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:08.820014] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:09.820268] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:09.827822] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:10.828244] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:10.837254] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:11.837557] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:11.850650] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:12.851145] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:12.859275] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:13.859549] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:13.868077] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:14.868359] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:14.877077] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:15.877400] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:15.889237] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:16.889498] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:16.897371] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:17.897639] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:17.906256] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:18.906521] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:18.915391] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:19.915650] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:19.923802] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:20.924254] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:20.934622] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:21.934898] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:21.943449] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:22.943712] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:22.952258] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:23.952638] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:23.960729] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:24.961128] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:24.969350] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:25.969646] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:25.977512] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:26.977793] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:26.986036] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:27.986500] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:27.995532] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:28.995953] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:29.005572] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:30.005938] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:30.014554] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:31.014961] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:31.026863] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:32.027218] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:32.035941] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:33.036276] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:33.044615] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:34.044949] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:34.054263] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:35.054580] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:35.062926] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:36.063227] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:36.076522] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:37.076970] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:37.085534] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:38.086004] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:38.148313] end - ✅ in 0.062s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:39.148834] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:39.156641] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:40.157024] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:40.166143] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:41.166410] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:41.177132] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:42.177416] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:42.186176] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:43.186436] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:43.194817] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:44.195235] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:44.203351] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:45.203668] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:45.212387] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:46.212711] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:46.221624] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:47.222153] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:47.232842] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:48.233366] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:48.241411] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:49.241697] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:49.250669] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:50.250951] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:50.259290] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:51.259592] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:51.267894] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:52.268264] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:52.276733] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:53.277211] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:53.285452] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:54.285732] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:54.293882] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:55.294393] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:55.303132] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:56.303593] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:56.311435] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:57.311730] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:57.321645] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:58.321970] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:58.343572] end - ✅ in 0.021s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:36:59.343891] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:36:59.354516] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:00.354995] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:00.363560] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:01.363963] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:01.372628] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:02.372968] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:02.381591] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:03.381987] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:03.390617] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:04.390955] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:04.398840] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:05.399160] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:05.407227] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:06.407634] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:06.415980] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:07.416265] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:07.424828] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:08.425201] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:08.441075] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:09.441342] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:09.449820] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:10.450155] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:10.458284] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:11.458592] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:11.465968] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:12.466273] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:12.474771] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:13.475275] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:13.483677] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:14.484193] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:14.492470] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:15.492760] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:15.500635] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:16.500939] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:16.508957] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:17.509254] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:17.517514] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:18.517866] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:18.525618] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:19.525951] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:19.533725] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:20.534070] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:20.542202] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:21.542489] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:21.550304] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:22.550562] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:22.558204] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:23.558488] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:23.567395] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:24.567665] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:24.575525] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:25.575800] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:25.584430] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:26.584707] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:26.592505] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:27.592775] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:27.601072] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:28.601363] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:28.609246] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:29.609599] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:29.618374] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:30.618827] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:30.628514] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:31.628948] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:31.637293] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:32.637741] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:32.646163] end - ✅ in 0.008s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-04-23T18:37:32.646281] end - ❌ 900.572s: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:32.646416] start - args=(, 'llmisvc-model-fb-opt-125m-route-6fa1027a', 'kserve-ci-e2e-test'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:32.653408] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-04-23T18:37:34.075950] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-6fa1027a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-with-gateway-ref-llmisvc-8a6d75cb'}, [e2e-llm-inference-service] {'name': 'router-with-managed-route-llmis-615ab0ab'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-52a2b82b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-e25babec'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-04-23T18:37:34.105428] end - ✅ in 0.029s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-04-23T18:37:34.105527] end - ❌ 902.219s: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:39Z', 'message': 'The following Gateways are not ready: [kserve-ci-e2e-test/router-gateway-1]', 'reason': 'GatewaysNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-scheduler-with-inline-config-workload-llmd-simulator] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] [e2e-llm-inference-service] def _new_conn(self) -> socket.socket: [e2e-llm-inference-service] """Establish a socket connection and set nodelay settings on it. [e2e-llm-inference-service] [e2e-llm-inference-service] :return: New socket connection. [e2e-llm-inference-service] """ [e2e-llm-inference-service] try: [e2e-llm-inference-service] > sock = connection.create_connection( [e2e-llm-inference-service] (self._dns_host, self.port), [e2e-llm-inference-service] self.timeout, [e2e-llm-inference-service] source_address=self.source_address, [e2e-llm-inference-service] socket_options=self.socket_options, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] address = ('openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', 80) [e2e-llm-inference-service] timeout = 60, source_address = None, socket_options = [(6, 1, 1)] [e2e-llm-inference-service] [e2e-llm-inference-service] def create_connection( [e2e-llm-inference-service] address: tuple[str, int], [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] source_address: tuple[str, int] | None = None, [e2e-llm-inference-service] socket_options: _TYPE_SOCKET_OPTIONS | None = None, [e2e-llm-inference-service] ) -> socket.socket: [e2e-llm-inference-service] """Connect to *address* and return the socket object. [e2e-llm-inference-service] [e2e-llm-inference-service] Convenience function. Connect to *address* (a 2-tuple ``(host, [e2e-llm-inference-service] port)``) and return the socket object. Passing the optional [e2e-llm-inference-service] *timeout* parameter will set the timeout on the socket instance [e2e-llm-inference-service] before attempting to connect. If no *timeout* is supplied, the [e2e-llm-inference-service] global default timeout setting returned by :func:`socket.getdefaulttimeout` [e2e-llm-inference-service] is used. If *source_address* is set it must be a tuple of (host, port) [e2e-llm-inference-service] for the socket to bind as a source address before making the connection. [e2e-llm-inference-service] An host of '' or port 0 tells the OS to use the default. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] host, port = address [e2e-llm-inference-service] if host.startswith("["): [e2e-llm-inference-service] host = host.strip("[]") [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Using the value from allowed_gai_family() in the context of getaddrinfo lets [e2e-llm-inference-service] # us select whether to work with IPv4 DNS records, IPv6 records, or both. [e2e-llm-inference-service] # The original create_connection function always returns all records. [e2e-llm-inference-service] family = allowed_gai_family() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] host.encode("idna") [e2e-llm-inference-service] except UnicodeError: [e2e-llm-inference-service] raise LocationParseError(f"'{host}', label empty or too long") from None [e2e-llm-inference-service] [e2e-llm-inference-service] > for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/connection.py:60: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] host = 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' [e2e-llm-inference-service] port = 80, family = [e2e-llm-inference-service] type = , proto = 0, flags = 0 [e2e-llm-inference-service] [e2e-llm-inference-service] def getaddrinfo(host, port, family=0, type=0, proto=0, flags=0): [e2e-llm-inference-service] """Resolve host and port into list of address info entries. [e2e-llm-inference-service] [e2e-llm-inference-service] Translate the host/port argument into a sequence of 5-tuples that contain [e2e-llm-inference-service] all the necessary arguments for creating a socket connected to that service. [e2e-llm-inference-service] host is a domain name, a string representation of an IPv4/v6 address or [e2e-llm-inference-service] None. port is a string service name such as 'http', a numeric port number or [e2e-llm-inference-service] None. By passing None as the value of host and port, you can pass NULL to [e2e-llm-inference-service] the underlying C API. [e2e-llm-inference-service] [e2e-llm-inference-service] The family, type and proto arguments can be optionally specified in order to [e2e-llm-inference-service] narrow the list of addresses returned. Passing zero as a value for each of [e2e-llm-inference-service] these arguments selects the full range of results. [e2e-llm-inference-service] """ [e2e-llm-inference-service] # We override this function since we want to translate the numeric family [e2e-llm-inference-service] # and socket type values to enum constants. [e2e-llm-inference-service] addrlist = [] [e2e-llm-inference-service] > for res in _socket.getaddrinfo(host, port, family, type, proto, flags): [e2e-llm-inference-service] E socket.gaierror: [Errno -2] Name or service not known [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/socket.py:974: gaierror [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False, err = None, clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] > response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:787: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] conn = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), chunked = False [e2e-llm-inference-service] response_conn = [e2e-llm-inference-service] preload_content = False, decode_content = False, enforce_content_length = True [e2e-llm-inference-service] [e2e-llm-inference-service] def _make_request( [e2e-llm-inference-service] self, [e2e-llm-inference-service] conn: BaseHTTPConnection, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | None = None, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] response_conn: BaseHTTPConnection | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] enforce_content_length: bool = True, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Perform a request on a given urllib connection object taken from our [e2e-llm-inference-service] pool. [e2e-llm-inference-service] [e2e-llm-inference-service] :param conn: [e2e-llm-inference-service] a connection from one of our connection pools [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] Pass ``None`` to retry until you receive a response. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param response_conn: [e2e-llm-inference-service] Set this to ``None`` if you will handle releasing the connection or [e2e-llm-inference-service] set the connection to have the response release it. [e2e-llm-inference-service] [e2e-llm-inference-service] :param preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded during construction. [e2e-llm-inference-service] [e2e-llm-inference-service] :param decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param enforce_content_length: [e2e-llm-inference-service] Enforce content length checking. Body returned by server must match [e2e-llm-inference-service] value of Content-Length header, if present. Otherwise, raise error. [e2e-llm-inference-service] """ [e2e-llm-inference-service] self.num_requests += 1 [e2e-llm-inference-service] [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] timeout_obj.start_connect() [e2e-llm-inference-service] conn.timeout = Timeout.resolve_default_timeout(timeout_obj.connect_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Trigger any extra validation we need to do. [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._validate_conn(conn) [e2e-llm-inference-service] except (SocketTimeout, BaseSSLError) as e: [e2e-llm-inference-service] self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # _validate_conn() starts the connection to an HTTPS proxy [e2e-llm-inference-service] # so we need to wrap errors with 'ProxyError' here too. [e2e-llm-inference-service] except ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] # If the connection didn't successfully connect to it's proxy [e2e-llm-inference-service] # then there [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, (OSError, NewConnectionError, TimeoutError, SSLError) [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] raise new_e [e2e-llm-inference-service] [e2e-llm-inference-service] # conn.request() calls http.client.*.request, not the method in [e2e-llm-inference-service] # urllib3.request. It also calls makefile (recv) on the socket. [e2e-llm-inference-service] try: [e2e-llm-inference-service] > conn.request( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] enforce_content_length=enforce_content_length, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:493: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] [e2e-llm-inference-service] def request( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] *, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] enforce_content_length: bool = True, [e2e-llm-inference-service] ) -> None: [e2e-llm-inference-service] # Update the inner socket's timeout value to send the request. [e2e-llm-inference-service] # This only triggers if the connection is re-used. [e2e-llm-inference-service] if self.sock is not None: [e2e-llm-inference-service] self.sock.settimeout(self.timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] # Store these values to be fed into the HTTPResponse [e2e-llm-inference-service] # object later. TODO: Remove this in favor of a real [e2e-llm-inference-service] # HTTP lifecycle mechanism. [e2e-llm-inference-service] [e2e-llm-inference-service] # We have to store these before we call .request() [e2e-llm-inference-service] # because sometimes we can still salvage a response [e2e-llm-inference-service] # off the wire even if we aren't able to completely [e2e-llm-inference-service] # send the request body. [e2e-llm-inference-service] self._response_options = _ResponseOptions( [e2e-llm-inference-service] request_method=method, [e2e-llm-inference-service] request_url=url, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] enforce_content_length=enforce_content_length, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = {} [e2e-llm-inference-service] header_keys = frozenset(to_str(k.lower()) for k in headers) [e2e-llm-inference-service] skip_accept_encoding = "accept-encoding" in header_keys [e2e-llm-inference-service] skip_host = "host" in header_keys [e2e-llm-inference-service] self.putrequest( [e2e-llm-inference-service] method, url, skip_accept_encoding=skip_accept_encoding, skip_host=skip_host [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Transform the body into an iterable of sendall()-able chunks [e2e-llm-inference-service] # and detect if an explicit Content-Length is doable. [e2e-llm-inference-service] chunks_and_cl = body_to_chunks(body, method=method, blocksize=self.blocksize) [e2e-llm-inference-service] chunks = chunks_and_cl.chunks [e2e-llm-inference-service] content_length = chunks_and_cl.content_length [e2e-llm-inference-service] [e2e-llm-inference-service] # When chunked is explicit set to 'True' we respect that. [e2e-llm-inference-service] if chunked: [e2e-llm-inference-service] if "transfer-encoding" not in header_keys: [e2e-llm-inference-service] self.putheader("Transfer-Encoding", "chunked") [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Detect whether a framing mechanism is already in use. If so [e2e-llm-inference-service] # we respect that value, otherwise we pick chunked vs content-length [e2e-llm-inference-service] # depending on the type of 'body'. [e2e-llm-inference-service] if "content-length" in header_keys: [e2e-llm-inference-service] chunked = False [e2e-llm-inference-service] elif "transfer-encoding" in header_keys: [e2e-llm-inference-service] chunked = True [e2e-llm-inference-service] [e2e-llm-inference-service] # Otherwise we go off the recommendation of 'body_to_chunks()'. [e2e-llm-inference-service] else: [e2e-llm-inference-service] chunked = False [e2e-llm-inference-service] if content_length is None: [e2e-llm-inference-service] if chunks is not None: [e2e-llm-inference-service] chunked = True [e2e-llm-inference-service] self.putheader("Transfer-Encoding", "chunked") [e2e-llm-inference-service] else: [e2e-llm-inference-service] self.putheader("Content-Length", str(content_length)) [e2e-llm-inference-service] [e2e-llm-inference-service] # Now that framing headers are out of the way we send all the other headers. [e2e-llm-inference-service] if "user-agent" not in header_keys: [e2e-llm-inference-service] self.putheader("User-Agent", _get_default_user_agent()) [e2e-llm-inference-service] for header, value in headers.items(): [e2e-llm-inference-service] self.putheader(header, value) [e2e-llm-inference-service] > self.endheaders() [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:500: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] message_body = None [e2e-llm-inference-service] [e2e-llm-inference-service] def endheaders(self, message_body=None, *, encode_chunked=False): [e2e-llm-inference-service] """Indicate that the last header line has been sent to the server. [e2e-llm-inference-service] [e2e-llm-inference-service] This method sends the request to the server. The optional message_body [e2e-llm-inference-service] argument can be used to pass a message body associated with the [e2e-llm-inference-service] request. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if self.__state == _CS_REQ_STARTED: [e2e-llm-inference-service] self.__state = _CS_REQ_SENT [e2e-llm-inference-service] else: [e2e-llm-inference-service] raise CannotSendHeader() [e2e-llm-inference-service] > self._send_output(message_body, encode_chunked=encode_chunked) [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/http/client.py:1298: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] message_body = None, encode_chunked = False [e2e-llm-inference-service] [e2e-llm-inference-service] def _send_output(self, message_body=None, encode_chunked=False): [e2e-llm-inference-service] """Send the currently buffered request and clear the buffer. [e2e-llm-inference-service] [e2e-llm-inference-service] Appends an extra \\r\\n to the buffer. [e2e-llm-inference-service] A message_body may be specified, to be appended to the request. [e2e-llm-inference-service] """ [e2e-llm-inference-service] self._buffer.extend((b"", b"")) [e2e-llm-inference-service] msg = b"\r\n".join(self._buffer) [e2e-llm-inference-service] del self._buffer[:] [e2e-llm-inference-service] > self.send(msg) [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/http/client.py:1058: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] data = b'POST /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions HTTP/1.1\r\nHost: openshift-ai-inference-opensh...gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Type: application/json\r\nContent-Length: 74\r\n\r\n' [e2e-llm-inference-service] [e2e-llm-inference-service] def send(self, data): [e2e-llm-inference-service] """Send `data' to the server. [e2e-llm-inference-service] ``data`` can be a string object, a bytes object, an array object, a [e2e-llm-inference-service] file-like object that supports a .read() method, or an iterable object. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] if self.sock is None: [e2e-llm-inference-service] if self.auto_open: [e2e-llm-inference-service] > self.connect() [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/http/client.py:996: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] [e2e-llm-inference-service] def connect(self) -> None: [e2e-llm-inference-service] > self.sock = self._new_conn() [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:331: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] [e2e-llm-inference-service] def _new_conn(self) -> socket.socket: [e2e-llm-inference-service] """Establish a socket connection and set nodelay settings on it. [e2e-llm-inference-service] [e2e-llm-inference-service] :return: New socket connection. [e2e-llm-inference-service] """ [e2e-llm-inference-service] try: [e2e-llm-inference-service] sock = connection.create_connection( [e2e-llm-inference-service] (self._dns_host, self.port), [e2e-llm-inference-service] self.timeout, [e2e-llm-inference-service] source_address=self.source_address, [e2e-llm-inference-service] socket_options=self.socket_options, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except socket.gaierror as e: [e2e-llm-inference-service] > raise NameResolutionError(self.host, self, e) from e [e2e-llm-inference-service] E urllib3.exceptions.NameResolutionError: HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:211: NameResolutionError [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] request = , stream = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), verify = '/tmp/ca.crt' [e2e-llm-inference-service] cert = None, proxies = OrderedDict() [e2e-llm-inference-service] [e2e-llm-inference-service] def send( [e2e-llm-inference-service] self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Sends PreparedRequest object. Returns Response object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param request: The :class:`PreparedRequest ` being sent. [e2e-llm-inference-service] :param stream: (optional) Whether to stream the request content. [e2e-llm-inference-service] :param timeout: (optional) How long to wait for the server to send [e2e-llm-inference-service] data before giving up, as a float, or a :ref:`(connect timeout, [e2e-llm-inference-service] read timeout) ` tuple. [e2e-llm-inference-service] :type timeout: float or tuple or urllib3 Timeout object [e2e-llm-inference-service] :param verify: (optional) Either a boolean, in which case it controls whether [e2e-llm-inference-service] we verify the server's TLS certificate, or a string, in which case it [e2e-llm-inference-service] must be a path to a CA bundle to use [e2e-llm-inference-service] :param cert: (optional) Any user-provided SSL certificate to be trusted. [e2e-llm-inference-service] :param proxies: (optional) The proxies dictionary to apply to the request. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] conn = self.get_connection_with_tls_context( [e2e-llm-inference-service] request, verify, proxies=proxies, cert=cert [e2e-llm-inference-service] ) [e2e-llm-inference-service] except LocationValueError as e: [e2e-llm-inference-service] raise InvalidURL(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] self.cert_verify(conn, request.url, verify, cert) [e2e-llm-inference-service] url = self.request_url(request, proxies) [e2e-llm-inference-service] self.add_headers( [e2e-llm-inference-service] request, [e2e-llm-inference-service] stream=stream, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] verify=verify, [e2e-llm-inference-service] cert=cert, [e2e-llm-inference-service] proxies=proxies, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] chunked = not (request.body is None or "Content-Length" in request.headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(timeout, tuple): [e2e-llm-inference-service] try: [e2e-llm-inference-service] connect, read = timeout [e2e-llm-inference-service] timeout = TimeoutSauce(connect=connect, read=read) [e2e-llm-inference-service] except ValueError: [e2e-llm-inference-service] raise ValueError( [e2e-llm-inference-service] f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, " [e2e-llm-inference-service] f"or a single float to set both timeouts to the same value." [e2e-llm-inference-service] ) [e2e-llm-inference-service] elif isinstance(timeout, TimeoutSauce): [e2e-llm-inference-service] pass [e2e-llm-inference-service] else: [e2e-llm-inference-service] timeout = TimeoutSauce(connect=timeout, read=timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > resp = conn.urlopen( [e2e-llm-inference-service] method=request.method, [e2e-llm-inference-service] url=url, [e2e-llm-inference-service] body=request.body, [e2e-llm-inference-service] headers=request.headers, [e2e-llm-inference-service] redirect=False, [e2e-llm-inference-service] assert_same_host=False, [e2e-llm-inference-service] preload_content=False, [e2e-llm-inference-service] decode_content=False, [e2e-llm-inference-service] retries=self.max_retries, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/adapters.py:667: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=7, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=6, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=5, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=4, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=3, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=2, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=1, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False, err = None, clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] > retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:841: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] response = None [e2e-llm-inference-service] error = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] _pool = [e2e-llm-inference-service] _stacktrace = [e2e-llm-inference-service] [e2e-llm-inference-service] def increment( [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str | None = None, [e2e-llm-inference-service] url: str | None = None, [e2e-llm-inference-service] response: BaseHTTPResponse | None = None, [e2e-llm-inference-service] error: Exception | None = None, [e2e-llm-inference-service] _pool: ConnectionPool | None = None, [e2e-llm-inference-service] _stacktrace: TracebackType | None = None, [e2e-llm-inference-service] ) -> Self: [e2e-llm-inference-service] """Return a new Retry object with incremented retry counters. [e2e-llm-inference-service] [e2e-llm-inference-service] :param response: A response object, or None, if the server did not [e2e-llm-inference-service] return a response. [e2e-llm-inference-service] :type response: :class:`~urllib3.response.BaseHTTPResponse` [e2e-llm-inference-service] :param Exception error: An error encountered during the request, or [e2e-llm-inference-service] None if the response was received successfully. [e2e-llm-inference-service] [e2e-llm-inference-service] :return: A new ``Retry`` object. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if self.total is False and error: [e2e-llm-inference-service] # Disabled, indicate to re-raise the error. [e2e-llm-inference-service] raise reraise(type(error), error, _stacktrace) [e2e-llm-inference-service] [e2e-llm-inference-service] total = self.total [e2e-llm-inference-service] if total is not None: [e2e-llm-inference-service] total -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] connect = self.connect [e2e-llm-inference-service] read = self.read [e2e-llm-inference-service] redirect = self.redirect [e2e-llm-inference-service] status_count = self.status [e2e-llm-inference-service] other = self.other [e2e-llm-inference-service] cause = "unknown" [e2e-llm-inference-service] status = None [e2e-llm-inference-service] redirect_location = None [e2e-llm-inference-service] [e2e-llm-inference-service] if error and self._is_connection_error(error): [e2e-llm-inference-service] # Connect retry? [e2e-llm-inference-service] if connect is False: [e2e-llm-inference-service] raise reraise(type(error), error, _stacktrace) [e2e-llm-inference-service] elif connect is not None: [e2e-llm-inference-service] connect -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] elif error and self._is_read_error(error): [e2e-llm-inference-service] # Read retry? [e2e-llm-inference-service] if read is False or method is None or not self._is_method_retryable(method): [e2e-llm-inference-service] raise reraise(type(error), error, _stacktrace) [e2e-llm-inference-service] elif read is not None: [e2e-llm-inference-service] read -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] elif error: [e2e-llm-inference-service] # Other retry? [e2e-llm-inference-service] if other is not None: [e2e-llm-inference-service] other -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] elif response and response.get_redirect_location(): [e2e-llm-inference-service] # Redirect retry? [e2e-llm-inference-service] if redirect is not None: [e2e-llm-inference-service] redirect -= 1 [e2e-llm-inference-service] cause = "too many redirects" [e2e-llm-inference-service] response_redirect_location = response.get_redirect_location() [e2e-llm-inference-service] if response_redirect_location: [e2e-llm-inference-service] redirect_location = response_redirect_location [e2e-llm-inference-service] status = response.status [e2e-llm-inference-service] [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Incrementing because of a server error like a 500 in [e2e-llm-inference-service] # status_forcelist and the given method is in the allowed_methods [e2e-llm-inference-service] cause = ResponseError.GENERIC_ERROR [e2e-llm-inference-service] if response and response.status: [e2e-llm-inference-service] if status_count is not None: [e2e-llm-inference-service] status_count -= 1 [e2e-llm-inference-service] cause = ResponseError.SPECIFIC_ERROR.format(status_code=response.status) [e2e-llm-inference-service] status = response.status [e2e-llm-inference-service] [e2e-llm-inference-service] history = self.history + ( [e2e-llm-inference-service] RequestHistory(method, url, error, status, redirect_location), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] new_retry = self.new( [e2e-llm-inference-service] total=total, [e2e-llm-inference-service] connect=connect, [e2e-llm-inference-service] read=read, [e2e-llm-inference-service] redirect=redirect, [e2e-llm-inference-service] status=status_count, [e2e-llm-inference-service] other=other, [e2e-llm-inference-service] history=history, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if new_retry.is_exhausted(): [e2e-llm-inference-service] reason = error or ResponseError(cause) [e2e-llm-inference-service] > raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] [e2e-llm-inference-service] E urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/retry.py:519: MaxRetryError [e2e-llm-inference-service] [e2e-llm-inference-service] During handling of the above exception, another exception occurred: [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_model_responds(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name, [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] > response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:520: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] url = 'http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] [e2e-llm-inference-service] def post_with_retry( [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] *, [e2e-llm-inference-service] headers: Dict = None, [e2e-llm-inference-service] json_data: Union[Dict, List] = None, [e2e-llm-inference-service] data: Union[str, bytes] = None, [e2e-llm-inference-service] stream: bool = False, [e2e-llm-inference-service] timeout: float = None, [e2e-llm-inference-service] total_retries: int = DEFAULT_RETRY_TOTAL, [e2e-llm-inference-service] backoff_factor: float = DEFAULT_RETRY_BACKOFF_FACTOR, [e2e-llm-inference-service] retry_status_codes=DEFAULT_RETRY_STATUS_CODES, [e2e-llm-inference-service] ) -> requests.Response: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Send POST request with retries for transient HTTP and network failures. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if json_data is not None and data is not None: [e2e-llm-inference-service] raise ValueError("Only one of json_data or data can be provided.") [e2e-llm-inference-service] [e2e-llm-inference-service] retry = Retry( [e2e-llm-inference-service] total=total_retries, [e2e-llm-inference-service] backoff_factor=backoff_factor, [e2e-llm-inference-service] status_forcelist=retry_status_codes, [e2e-llm-inference-service] allowed_methods=["POST"], [e2e-llm-inference-service] raise_on_status=False, [e2e-llm-inference-service] ) [e2e-llm-inference-service] with requests.Session() as session: [e2e-llm-inference-service] session.mount("http://", HTTPAdapter(max_retries=retry)) [e2e-llm-inference-service] session.mount("https://", HTTPAdapter(max_retries=retry)) [e2e-llm-inference-service] > return session.post( [e2e-llm-inference-service] url, [e2e-llm-inference-service] json=json_data, [e2e-llm-inference-service] data=data, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] stream=stream, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] common/http_retry.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] data = None [e2e-llm-inference-service] json = {'max_tokens': 100, 'model': 'facebook/opt-125m', 'prompt': 'KServe is a'} [e2e-llm-inference-service] kwargs = {'headers': {'Content-Type': 'application/json'}, 'stream': False, 'timeout': 60} [e2e-llm-inference-service] [e2e-llm-inference-service] def post(self, url, data=None, json=None, **kwargs): [e2e-llm-inference-service] r"""Sends a POST request. Returns :class:`Response` object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: URL for the new :class:`Request` object. [e2e-llm-inference-service] :param data: (optional) Dictionary, list of tuples, bytes, or file-like [e2e-llm-inference-service] object to send in the body of the :class:`Request`. [e2e-llm-inference-service] :param json: (optional) json to send in the body of the :class:`Request`. [e2e-llm-inference-service] :param \*\*kwargs: Optional arguments that ``request`` takes. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.request("POST", url, data=data, json=json, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/sessions.py:637: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = , method = 'POST' [e2e-llm-inference-service] url = 'http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions' [e2e-llm-inference-service] params = None, data = None, headers = {'Content-Type': 'application/json'} [e2e-llm-inference-service] cookies = None, files = None, auth = None, timeout = 60, allow_redirects = True [e2e-llm-inference-service] proxies = {}, hooks = None, stream = False, verify = None, cert = None [e2e-llm-inference-service] json = {'max_tokens': 100, 'model': 'facebook/opt-125m', 'prompt': 'KServe is a'} [e2e-llm-inference-service] [e2e-llm-inference-service] def request( [e2e-llm-inference-service] self, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] params=None, [e2e-llm-inference-service] data=None, [e2e-llm-inference-service] headers=None, [e2e-llm-inference-service] cookies=None, [e2e-llm-inference-service] files=None, [e2e-llm-inference-service] auth=None, [e2e-llm-inference-service] timeout=None, [e2e-llm-inference-service] allow_redirects=True, [e2e-llm-inference-service] proxies=None, [e2e-llm-inference-service] hooks=None, [e2e-llm-inference-service] stream=None, [e2e-llm-inference-service] verify=None, [e2e-llm-inference-service] cert=None, [e2e-llm-inference-service] json=None, [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Constructs a :class:`Request `, prepares it and sends it. [e2e-llm-inference-service] Returns :class:`Response ` object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: method for the new :class:`Request` object. [e2e-llm-inference-service] :param url: URL for the new :class:`Request` object. [e2e-llm-inference-service] :param params: (optional) Dictionary or bytes to be sent in the query [e2e-llm-inference-service] string for the :class:`Request`. [e2e-llm-inference-service] :param data: (optional) Dictionary, list of tuples, bytes, or file-like [e2e-llm-inference-service] object to send in the body of the :class:`Request`. [e2e-llm-inference-service] :param json: (optional) json to send in the body of the [e2e-llm-inference-service] :class:`Request`. [e2e-llm-inference-service] :param headers: (optional) Dictionary of HTTP Headers to send with the [e2e-llm-inference-service] :class:`Request`. [e2e-llm-inference-service] :param cookies: (optional) Dict or CookieJar object to send with the [e2e-llm-inference-service] :class:`Request`. [e2e-llm-inference-service] :param files: (optional) Dictionary of ``'filename': file-like-objects`` [e2e-llm-inference-service] for multipart encoding upload. [e2e-llm-inference-service] :param auth: (optional) Auth tuple or callable to enable [e2e-llm-inference-service] Basic/Digest/Custom HTTP Auth. [e2e-llm-inference-service] :param timeout: (optional) How long to wait for the server to send [e2e-llm-inference-service] data before giving up, as a float, or a :ref:`(connect timeout, [e2e-llm-inference-service] read timeout) ` tuple. [e2e-llm-inference-service] :type timeout: float or tuple [e2e-llm-inference-service] :param allow_redirects: (optional) Set to True by default. [e2e-llm-inference-service] :type allow_redirects: bool [e2e-llm-inference-service] :param proxies: (optional) Dictionary mapping protocol or protocol and [e2e-llm-inference-service] hostname to the URL of the proxy. [e2e-llm-inference-service] :param hooks: (optional) Dictionary mapping hook name to one event or [e2e-llm-inference-service] list of events, event must be callable. [e2e-llm-inference-service] :param stream: (optional) whether to immediately download the response [e2e-llm-inference-service] content. Defaults to ``False``. [e2e-llm-inference-service] :param verify: (optional) Either a boolean, in which case it controls whether we verify [e2e-llm-inference-service] the server's TLS certificate, or a string, in which case it must be a path [e2e-llm-inference-service] to a CA bundle to use. Defaults to ``True``. When set to [e2e-llm-inference-service] ``False``, requests will accept any TLS certificate presented by [e2e-llm-inference-service] the server, and will ignore hostname mismatches and/or expired [e2e-llm-inference-service] certificates, which will make your application vulnerable to [e2e-llm-inference-service] man-in-the-middle (MitM) attacks. Setting verify to ``False`` [e2e-llm-inference-service] may be useful during local development or testing. [e2e-llm-inference-service] :param cert: (optional) if String, path to ssl client cert file (.pem). [e2e-llm-inference-service] If Tuple, ('cert', 'key') pair. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] # Create the Request. [e2e-llm-inference-service] req = Request( [e2e-llm-inference-service] method=method.upper(), [e2e-llm-inference-service] url=url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] files=files, [e2e-llm-inference-service] data=data or {}, [e2e-llm-inference-service] json=json, [e2e-llm-inference-service] params=params or {}, [e2e-llm-inference-service] auth=auth, [e2e-llm-inference-service] cookies=cookies, [e2e-llm-inference-service] hooks=hooks, [e2e-llm-inference-service] ) [e2e-llm-inference-service] prep = self.prepare_request(req) [e2e-llm-inference-service] [e2e-llm-inference-service] proxies = proxies or {} [e2e-llm-inference-service] [e2e-llm-inference-service] settings = self.merge_environment_settings( [e2e-llm-inference-service] prep.url, proxies, stream, verify, cert [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Send the request. [e2e-llm-inference-service] send_kwargs = { [e2e-llm-inference-service] "timeout": timeout, [e2e-llm-inference-service] "allow_redirects": allow_redirects, [e2e-llm-inference-service] } [e2e-llm-inference-service] send_kwargs.update(settings) [e2e-llm-inference-service] > resp = self.send(prep, **send_kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/sessions.py:589: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] request = [e2e-llm-inference-service] kwargs = {'cert': None, 'proxies': OrderedDict(), 'stream': False, 'timeout': 60, ...} [e2e-llm-inference-service] allow_redirects = True, stream = False, hooks = {'response': []} [e2e-llm-inference-service] adapter = [e2e-llm-inference-service] start = 1776969291.502064 [e2e-llm-inference-service] [e2e-llm-inference-service] def send(self, request, **kwargs): [e2e-llm-inference-service] """Send a given PreparedRequest. [e2e-llm-inference-service] [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] # Set defaults that the hooks can utilize to ensure they always have [e2e-llm-inference-service] # the correct parameters to reproduce the previous request. [e2e-llm-inference-service] kwargs.setdefault("stream", self.stream) [e2e-llm-inference-service] kwargs.setdefault("verify", self.verify) [e2e-llm-inference-service] kwargs.setdefault("cert", self.cert) [e2e-llm-inference-service] if "proxies" not in kwargs: [e2e-llm-inference-service] kwargs["proxies"] = resolve_proxies(request, self.proxies, self.trust_env) [e2e-llm-inference-service] [e2e-llm-inference-service] # It's possible that users might accidentally send a Request object. [e2e-llm-inference-service] # Guard against that specific failure case. [e2e-llm-inference-service] if isinstance(request, Request): [e2e-llm-inference-service] raise ValueError("You can only send PreparedRequests.") [e2e-llm-inference-service] [e2e-llm-inference-service] # Set up variables needed for resolve_redirects and dispatching of hooks [e2e-llm-inference-service] allow_redirects = kwargs.pop("allow_redirects", True) [e2e-llm-inference-service] stream = kwargs.get("stream") [e2e-llm-inference-service] hooks = request.hooks [e2e-llm-inference-service] [e2e-llm-inference-service] # Get the appropriate adapter to use [e2e-llm-inference-service] adapter = self.get_adapter(url=request.url) [e2e-llm-inference-service] [e2e-llm-inference-service] # Start time (approximately) of the request [e2e-llm-inference-service] start = preferred_clock() [e2e-llm-inference-service] [e2e-llm-inference-service] # Send the request [e2e-llm-inference-service] > r = adapter.send(request, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/sessions.py:703: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] request = , stream = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), verify = '/tmp/ca.crt' [e2e-llm-inference-service] cert = None, proxies = OrderedDict() [e2e-llm-inference-service] [e2e-llm-inference-service] def send( [e2e-llm-inference-service] self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Sends PreparedRequest object. Returns Response object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param request: The :class:`PreparedRequest ` being sent. [e2e-llm-inference-service] :param stream: (optional) Whether to stream the request content. [e2e-llm-inference-service] :param timeout: (optional) How long to wait for the server to send [e2e-llm-inference-service] data before giving up, as a float, or a :ref:`(connect timeout, [e2e-llm-inference-service] read timeout) ` tuple. [e2e-llm-inference-service] :type timeout: float or tuple or urllib3 Timeout object [e2e-llm-inference-service] :param verify: (optional) Either a boolean, in which case it controls whether [e2e-llm-inference-service] we verify the server's TLS certificate, or a string, in which case it [e2e-llm-inference-service] must be a path to a CA bundle to use [e2e-llm-inference-service] :param cert: (optional) Any user-provided SSL certificate to be trusted. [e2e-llm-inference-service] :param proxies: (optional) The proxies dictionary to apply to the request. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] conn = self.get_connection_with_tls_context( [e2e-llm-inference-service] request, verify, proxies=proxies, cert=cert [e2e-llm-inference-service] ) [e2e-llm-inference-service] except LocationValueError as e: [e2e-llm-inference-service] raise InvalidURL(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] self.cert_verify(conn, request.url, verify, cert) [e2e-llm-inference-service] url = self.request_url(request, proxies) [e2e-llm-inference-service] self.add_headers( [e2e-llm-inference-service] request, [e2e-llm-inference-service] stream=stream, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] verify=verify, [e2e-llm-inference-service] cert=cert, [e2e-llm-inference-service] proxies=proxies, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] chunked = not (request.body is None or "Content-Length" in request.headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(timeout, tuple): [e2e-llm-inference-service] try: [e2e-llm-inference-service] connect, read = timeout [e2e-llm-inference-service] timeout = TimeoutSauce(connect=connect, read=read) [e2e-llm-inference-service] except ValueError: [e2e-llm-inference-service] raise ValueError( [e2e-llm-inference-service] f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, " [e2e-llm-inference-service] f"or a single float to set both timeouts to the same value." [e2e-llm-inference-service] ) [e2e-llm-inference-service] elif isinstance(timeout, TimeoutSauce): [e2e-llm-inference-service] pass [e2e-llm-inference-service] else: [e2e-llm-inference-service] timeout = TimeoutSauce(connect=timeout, read=timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] resp = conn.urlopen( [e2e-llm-inference-service] method=request.method, [e2e-llm-inference-service] url=url, [e2e-llm-inference-service] body=request.body, [e2e-llm-inference-service] headers=request.headers, [e2e-llm-inference-service] redirect=False, [e2e-llm-inference-service] assert_same_host=False, [e2e-llm-inference-service] preload_content=False, [e2e-llm-inference-service] decode_content=False, [e2e-llm-inference-service] retries=self.max_retries, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] except (ProtocolError, OSError) as err: [e2e-llm-inference-service] raise ConnectionError(err, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] except MaxRetryError as e: [e2e-llm-inference-service] if isinstance(e.reason, ConnectTimeoutError): [e2e-llm-inference-service] # TODO: Remove this in 3.0.0: see #2811 [e2e-llm-inference-service] if not isinstance(e.reason, NewConnectionError): [e2e-llm-inference-service] raise ConnectTimeout(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(e.reason, ResponseError): [e2e-llm-inference-service] raise RetryError(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(e.reason, _ProxyError): [e2e-llm-inference-service] raise ProxyError(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(e.reason, _SSLError): [e2e-llm-inference-service] # This branch is for urllib3 v1.22 and later. [e2e-llm-inference-service] raise SSLError(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] > raise ConnectionError(e, request=request) [e2e-llm-inference-service] E requests.exceptions.ConnectionError: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/adapters.py:700: ConnectionError [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-with-inline-config', 'workload-llmd-simulator'], prompt='KServe is a'... {'name': 'workload-llmd-simulator-schedul-39bca5db'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] > wait_for_model_response(kserve_client, test_case, test_case.wait_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:413: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'scheduler-wi... {'name': 'workload-llmd-simulator-schedul-39bca5db'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-04-23T18:22:33.074414', start_time = 1776968553.0747128 [e2e-llm-inference-service] duration = 1102.5422387123108, timestamp_end = '2026-04-23T18:40:55.616955' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-with-inline-config', 'workload-llmd-simulator'], prompt='KServe is a'... {'name': 'workload-llmd-simulator-schedul-39bca5db'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900, extra_headers = None [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_model_responds(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name, [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.response_assertion(response) [e2e-llm-inference-service] return response.text[: test_case.max_tokens] [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_model_responds, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:535: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_model_responds at 0x7fb8efe9e0c0> [e2e-llm-inference-service] timeout = 900, interval = 5.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:628: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_model_responds(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name, [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] > raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] E AssertionError: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:528: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO kserve:conftest.py:40 Logger configured [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1107 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1056 Checking LLMInferenceServiceConfig router-managed-scheduler-inline-23288697 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1082 Resource not found, creating LLMInferenceServiceConfig router-managed-scheduler-inline-23288697 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1092 ✓ Successfully created LLMInferenceServiceConfig router-managed-scheduler-inline-23288697 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1056 Checking LLMInferenceServiceConfig scheduler-with-inline-config-sc-9914b4fd in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1082 Resource not found, creating LLMInferenceServiceConfig scheduler-with-inline-config-sc-9914b4fd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1092 ✓ Successfully created LLMInferenceServiceConfig scheduler-with-inline-config-sc-9914b4fd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1056 Checking LLMInferenceServiceConfig workload-llmd-simulator-schedul-39bca5db in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1082 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-schedul-39bca5db [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1092 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-schedul-39bca5db [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-04-23T18:21:31.987749] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'scheduler-with-inline-config', 'workload-llmd-simulator'], prompt='KServe is a', service_name='scheduler-inline-config-test', endpoint='/v1/completions', max_tokens=100, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, before_test=[], after_test=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-inline-config-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-inline-23288697'}, [e2e-llm-inference-service] {'name': 'scheduler-with-inline-config-sc-9914b4fd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-39bca5db'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1107 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-04-23T18:21:32.000291] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-inline-config-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-inline-23288697'}, [e2e-llm-inference-service] {'name': 'scheduler-with-inline-config-sc-9914b4fd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-39bca5db'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-04-23T18:21:32.079722] end - ✅ in 0.079s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-04-23T18:21:32.079894] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-inline-config-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-inline-23288697'}, [e2e-llm-inference-service] {'name': 'scheduler-with-inline-config-sc-9914b4fd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-39bca5db'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:32.080196] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:32.085846] end - ✅ in 0.006s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:33.086290] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:33.173536] end - ✅ in 0.087s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:34.173990] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:34.184044] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:35.184299] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:35.192113] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:36.192373] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:36.200999] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:37.201321] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:37.273818] end - ✅ in 0.072s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:38.274230] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:38.281273] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:39.281567] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:39.288845] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:40.289232] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:40.373662] end - ✅ in 0.084s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:41.373953] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:41.383572] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:42.383881] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:42.474222] end - ✅ in 0.090s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:43.474562] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:43.482506] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:44.482806] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:44.491218] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:45.491499] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:45.574184] end - ✅ in 0.082s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:46.574514] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:46.583159] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:47.583450] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:47.591520] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:48.591803] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:48.599139] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:49.599426] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:49.607579] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:50.607934] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:50.673752] end - ✅ in 0.066s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:51.674137] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:51.681430] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:52.681718] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:52.689063] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:53.689346] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:53.773894] end - ✅ in 0.084s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:54.774245] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:54.874873] end - ✅ in 0.100s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:55.875273] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:55.973795] end - ✅ in 0.098s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:56.974164] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:57.173973] end - ✅ in 0.200s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:58.174305] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:58.182574] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:21:59.182887] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:21:59.190546] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:00.190864] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:00.273674] end - ✅ in 0.083s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:01.274055] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:01.282415] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:02.282688] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:02.292430] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:03.292843] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:03.301113] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:04.301396] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:04.309764] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:05.310236] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:05.318255] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:06.318526] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:06.326807] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:07.327176] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:07.335102] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:08.335378] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:08.374038] end - ✅ in 0.038s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:09.374370] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:09.383183] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:10.383506] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:10.392091] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:11.392371] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:11.400392] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:12.400669] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:12.408405] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:13.408753] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:13.417654] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:14.417954] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:14.426441] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:15.426737] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:15.434730] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:16.435119] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:16.443199] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:17.443641] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:17.473989] end - ✅ in 0.030s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:18.474337] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:18.483124] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:19.483561] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:19.492300] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:20.492585] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:20.500771] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:21.501115] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:21.509355] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:22.509660] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:22.518248] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:23.518656] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:23.527404] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:24.527756] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:24.536575] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:25.536897] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:25.547696] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:26.548083] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:26.555951] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:27.556241] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:27.564023] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:28.564352] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:28.573086] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:29.573414] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:29.674572] end - ✅ in 0.101s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:30.674948] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:30.774768] end - ✅ in 0.100s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:31.775146] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:31.874969] end - ✅ in 0.100s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:21:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:22:16Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:32.875318] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:33.074207] end - ✅ in 0.199s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-04-23T18:22:33.074360] end - ✅ in 60.994s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-04-23T18:22:33.074414] start - args=(, TestCase(base_refs=['router-managed', 'scheduler-with-inline-config', 'workload-llmd-simulator'], prompt='KServe is a', service_name='scheduler-inline-config-test', endpoint='/v1/completions', max_tokens=100, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, before_test=[], after_test=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-inline-config-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-inline-23288697'}, [e2e-llm-inference-service] {'name': 'scheduler-with-inline-config-sc-9914b4fd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-39bca5db'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-04-23T18:22:33.074719] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-inline-config-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-inline-23288697'}, [e2e-llm-inference-service] {'name': 'scheduler-with-inline-config-sc-9914b4fd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-39bca5db'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:22:33.074954] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:22:33.173863] end - ✅ in 0.099s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-04-23T18:22:33.174053] end - ✅ in 0.099s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:518 Calling LLM service at http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 100} [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] ERROR e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:527 ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-04-23T18:28:42.293637] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-inline-config-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-inline-23288697'}, [e2e-llm-inference-service] {'name': 'scheduler-with-inline-config-sc-9914b4fd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-39bca5db'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:28:42.294203] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:28:42.333063] end - ✅ in 0.039s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-04-23T18:28:42.333213] end - ✅ in 0.039s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:518 Calling LLM service at http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 100} [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] ERROR e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:527 ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-04-23T18:34:51.446750] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-inline-config-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-inline-23288697'}, [e2e-llm-inference-service] {'name': 'scheduler-with-inline-config-sc-9914b4fd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-39bca5db'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:34:51.447390] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:34:51.500464] end - ✅ in 0.053s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-04-23T18:34:51.500645] end - ✅ in 0.053s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:518 Calling LLM service at http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 100} [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions [e2e-llm-inference-service] ERROR e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:527 ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-04-23T18:40:55.616955] end - ❌ 1102.542s: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:55.617055] start - args=(, 'scheduler-inline-config-test', 'kserve-ci-e2e-test'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:55.651538] end - ✅ in 0.034s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-04-23T18:40:56.708247] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-inline-config-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-inline-23288697'}, [e2e-llm-inference-service] {'name': 'scheduler-with-inline-config-sc-9914b4fd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-39bca5db'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-04-23T18:40:56.728981] end - ✅ in 0.020s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-04-23T18:40:56.729090] end - ❌ 1164.741s: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-inline-config-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-workload-single-cpu-model-fb-opt-125m] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] [e2e-llm-inference-service] def _new_conn(self) -> socket.socket: [e2e-llm-inference-service] """Establish a socket connection and set nodelay settings on it. [e2e-llm-inference-service] [e2e-llm-inference-service] :return: New socket connection. [e2e-llm-inference-service] """ [e2e-llm-inference-service] try: [e2e-llm-inference-service] > sock = connection.create_connection( [e2e-llm-inference-service] (self._dns_host, self.port), [e2e-llm-inference-service] self.timeout, [e2e-llm-inference-service] source_address=self.source_address, [e2e-llm-inference-service] socket_options=self.socket_options, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] address = ('openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', 80) [e2e-llm-inference-service] timeout = 60, source_address = None, socket_options = [(6, 1, 1)] [e2e-llm-inference-service] [e2e-llm-inference-service] def create_connection( [e2e-llm-inference-service] address: tuple[str, int], [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] source_address: tuple[str, int] | None = None, [e2e-llm-inference-service] socket_options: _TYPE_SOCKET_OPTIONS | None = None, [e2e-llm-inference-service] ) -> socket.socket: [e2e-llm-inference-service] """Connect to *address* and return the socket object. [e2e-llm-inference-service] [e2e-llm-inference-service] Convenience function. Connect to *address* (a 2-tuple ``(host, [e2e-llm-inference-service] port)``) and return the socket object. Passing the optional [e2e-llm-inference-service] *timeout* parameter will set the timeout on the socket instance [e2e-llm-inference-service] before attempting to connect. If no *timeout* is supplied, the [e2e-llm-inference-service] global default timeout setting returned by :func:`socket.getdefaulttimeout` [e2e-llm-inference-service] is used. If *source_address* is set it must be a tuple of (host, port) [e2e-llm-inference-service] for the socket to bind as a source address before making the connection. [e2e-llm-inference-service] An host of '' or port 0 tells the OS to use the default. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] host, port = address [e2e-llm-inference-service] if host.startswith("["): [e2e-llm-inference-service] host = host.strip("[]") [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Using the value from allowed_gai_family() in the context of getaddrinfo lets [e2e-llm-inference-service] # us select whether to work with IPv4 DNS records, IPv6 records, or both. [e2e-llm-inference-service] # The original create_connection function always returns all records. [e2e-llm-inference-service] family = allowed_gai_family() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] host.encode("idna") [e2e-llm-inference-service] except UnicodeError: [e2e-llm-inference-service] raise LocationParseError(f"'{host}', label empty or too long") from None [e2e-llm-inference-service] [e2e-llm-inference-service] > for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/connection.py:60: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] host = 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' [e2e-llm-inference-service] port = 80, family = [e2e-llm-inference-service] type = , proto = 0, flags = 0 [e2e-llm-inference-service] [e2e-llm-inference-service] def getaddrinfo(host, port, family=0, type=0, proto=0, flags=0): [e2e-llm-inference-service] """Resolve host and port into list of address info entries. [e2e-llm-inference-service] [e2e-llm-inference-service] Translate the host/port argument into a sequence of 5-tuples that contain [e2e-llm-inference-service] all the necessary arguments for creating a socket connected to that service. [e2e-llm-inference-service] host is a domain name, a string representation of an IPv4/v6 address or [e2e-llm-inference-service] None. port is a string service name such as 'http', a numeric port number or [e2e-llm-inference-service] None. By passing None as the value of host and port, you can pass NULL to [e2e-llm-inference-service] the underlying C API. [e2e-llm-inference-service] [e2e-llm-inference-service] The family, type and proto arguments can be optionally specified in order to [e2e-llm-inference-service] narrow the list of addresses returned. Passing zero as a value for each of [e2e-llm-inference-service] these arguments selects the full range of results. [e2e-llm-inference-service] """ [e2e-llm-inference-service] # We override this function since we want to translate the numeric family [e2e-llm-inference-service] # and socket type values to enum constants. [e2e-llm-inference-service] addrlist = [] [e2e-llm-inference-service] > for res in _socket.getaddrinfo(host, port, family, type, proto, flags): [e2e-llm-inference-service] E socket.gaierror: [Errno -2] Name or service not known [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/socket.py:974: gaierror [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False, err = None, clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] > response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:787: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] conn = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), chunked = False [e2e-llm-inference-service] response_conn = [e2e-llm-inference-service] preload_content = False, decode_content = False, enforce_content_length = True [e2e-llm-inference-service] [e2e-llm-inference-service] def _make_request( [e2e-llm-inference-service] self, [e2e-llm-inference-service] conn: BaseHTTPConnection, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | None = None, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] response_conn: BaseHTTPConnection | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] enforce_content_length: bool = True, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Perform a request on a given urllib connection object taken from our [e2e-llm-inference-service] pool. [e2e-llm-inference-service] [e2e-llm-inference-service] :param conn: [e2e-llm-inference-service] a connection from one of our connection pools [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] Pass ``None`` to retry until you receive a response. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param response_conn: [e2e-llm-inference-service] Set this to ``None`` if you will handle releasing the connection or [e2e-llm-inference-service] set the connection to have the response release it. [e2e-llm-inference-service] [e2e-llm-inference-service] :param preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded during construction. [e2e-llm-inference-service] [e2e-llm-inference-service] :param decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param enforce_content_length: [e2e-llm-inference-service] Enforce content length checking. Body returned by server must match [e2e-llm-inference-service] value of Content-Length header, if present. Otherwise, raise error. [e2e-llm-inference-service] """ [e2e-llm-inference-service] self.num_requests += 1 [e2e-llm-inference-service] [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] timeout_obj.start_connect() [e2e-llm-inference-service] conn.timeout = Timeout.resolve_default_timeout(timeout_obj.connect_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Trigger any extra validation we need to do. [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._validate_conn(conn) [e2e-llm-inference-service] except (SocketTimeout, BaseSSLError) as e: [e2e-llm-inference-service] self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # _validate_conn() starts the connection to an HTTPS proxy [e2e-llm-inference-service] # so we need to wrap errors with 'ProxyError' here too. [e2e-llm-inference-service] except ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] # If the connection didn't successfully connect to it's proxy [e2e-llm-inference-service] # then there [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, (OSError, NewConnectionError, TimeoutError, SSLError) [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] raise new_e [e2e-llm-inference-service] [e2e-llm-inference-service] # conn.request() calls http.client.*.request, not the method in [e2e-llm-inference-service] # urllib3.request. It also calls makefile (recv) on the socket. [e2e-llm-inference-service] try: [e2e-llm-inference-service] > conn.request( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] enforce_content_length=enforce_content_length, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:493: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] [e2e-llm-inference-service] def request( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] *, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] enforce_content_length: bool = True, [e2e-llm-inference-service] ) -> None: [e2e-llm-inference-service] # Update the inner socket's timeout value to send the request. [e2e-llm-inference-service] # This only triggers if the connection is re-used. [e2e-llm-inference-service] if self.sock is not None: [e2e-llm-inference-service] self.sock.settimeout(self.timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] # Store these values to be fed into the HTTPResponse [e2e-llm-inference-service] # object later. TODO: Remove this in favor of a real [e2e-llm-inference-service] # HTTP lifecycle mechanism. [e2e-llm-inference-service] [e2e-llm-inference-service] # We have to store these before we call .request() [e2e-llm-inference-service] # because sometimes we can still salvage a response [e2e-llm-inference-service] # off the wire even if we aren't able to completely [e2e-llm-inference-service] # send the request body. [e2e-llm-inference-service] self._response_options = _ResponseOptions( [e2e-llm-inference-service] request_method=method, [e2e-llm-inference-service] request_url=url, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] enforce_content_length=enforce_content_length, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = {} [e2e-llm-inference-service] header_keys = frozenset(to_str(k.lower()) for k in headers) [e2e-llm-inference-service] skip_accept_encoding = "accept-encoding" in header_keys [e2e-llm-inference-service] skip_host = "host" in header_keys [e2e-llm-inference-service] self.putrequest( [e2e-llm-inference-service] method, url, skip_accept_encoding=skip_accept_encoding, skip_host=skip_host [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Transform the body into an iterable of sendall()-able chunks [e2e-llm-inference-service] # and detect if an explicit Content-Length is doable. [e2e-llm-inference-service] chunks_and_cl = body_to_chunks(body, method=method, blocksize=self.blocksize) [e2e-llm-inference-service] chunks = chunks_and_cl.chunks [e2e-llm-inference-service] content_length = chunks_and_cl.content_length [e2e-llm-inference-service] [e2e-llm-inference-service] # When chunked is explicit set to 'True' we respect that. [e2e-llm-inference-service] if chunked: [e2e-llm-inference-service] if "transfer-encoding" not in header_keys: [e2e-llm-inference-service] self.putheader("Transfer-Encoding", "chunked") [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Detect whether a framing mechanism is already in use. If so [e2e-llm-inference-service] # we respect that value, otherwise we pick chunked vs content-length [e2e-llm-inference-service] # depending on the type of 'body'. [e2e-llm-inference-service] if "content-length" in header_keys: [e2e-llm-inference-service] chunked = False [e2e-llm-inference-service] elif "transfer-encoding" in header_keys: [e2e-llm-inference-service] chunked = True [e2e-llm-inference-service] [e2e-llm-inference-service] # Otherwise we go off the recommendation of 'body_to_chunks()'. [e2e-llm-inference-service] else: [e2e-llm-inference-service] chunked = False [e2e-llm-inference-service] if content_length is None: [e2e-llm-inference-service] if chunks is not None: [e2e-llm-inference-service] chunked = True [e2e-llm-inference-service] self.putheader("Transfer-Encoding", "chunked") [e2e-llm-inference-service] else: [e2e-llm-inference-service] self.putheader("Content-Length", str(content_length)) [e2e-llm-inference-service] [e2e-llm-inference-service] # Now that framing headers are out of the way we send all the other headers. [e2e-llm-inference-service] if "user-agent" not in header_keys: [e2e-llm-inference-service] self.putheader("User-Agent", _get_default_user_agent()) [e2e-llm-inference-service] for header, value in headers.items(): [e2e-llm-inference-service] self.putheader(header, value) [e2e-llm-inference-service] > self.endheaders() [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:500: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] message_body = None [e2e-llm-inference-service] [e2e-llm-inference-service] def endheaders(self, message_body=None, *, encode_chunked=False): [e2e-llm-inference-service] """Indicate that the last header line has been sent to the server. [e2e-llm-inference-service] [e2e-llm-inference-service] This method sends the request to the server. The optional message_body [e2e-llm-inference-service] argument can be used to pass a message body associated with the [e2e-llm-inference-service] request. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if self.__state == _CS_REQ_STARTED: [e2e-llm-inference-service] self.__state = _CS_REQ_SENT [e2e-llm-inference-service] else: [e2e-llm-inference-service] raise CannotSendHeader() [e2e-llm-inference-service] > self._send_output(message_body, encode_chunked=encode_chunked) [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/http/client.py:1298: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] message_body = None, encode_chunked = False [e2e-llm-inference-service] [e2e-llm-inference-service] def _send_output(self, message_body=None, encode_chunked=False): [e2e-llm-inference-service] """Send the currently buffered request and clear the buffer. [e2e-llm-inference-service] [e2e-llm-inference-service] Appends an extra \\r\\n to the buffer. [e2e-llm-inference-service] A message_body may be specified, to be appended to the request. [e2e-llm-inference-service] """ [e2e-llm-inference-service] self._buffer.extend((b"", b"")) [e2e-llm-inference-service] msg = b"\r\n".join(self._buffer) [e2e-llm-inference-service] del self._buffer[:] [e2e-llm-inference-service] > self.send(msg) [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/http/client.py:1058: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] data = b'POST /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions HTTP/1.1\r\nHost: openshift-ai-infe...gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Type: application/json\r\nContent-Length: 74\r\n\r\n' [e2e-llm-inference-service] [e2e-llm-inference-service] def send(self, data): [e2e-llm-inference-service] """Send `data' to the server. [e2e-llm-inference-service] ``data`` can be a string object, a bytes object, an array object, a [e2e-llm-inference-service] file-like object that supports a .read() method, or an iterable object. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] if self.sock is None: [e2e-llm-inference-service] if self.auto_open: [e2e-llm-inference-service] > self.connect() [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/http/client.py:996: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] [e2e-llm-inference-service] def connect(self) -> None: [e2e-llm-inference-service] > self.sock = self._new_conn() [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:331: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] [e2e-llm-inference-service] def _new_conn(self) -> socket.socket: [e2e-llm-inference-service] """Establish a socket connection and set nodelay settings on it. [e2e-llm-inference-service] [e2e-llm-inference-service] :return: New socket connection. [e2e-llm-inference-service] """ [e2e-llm-inference-service] try: [e2e-llm-inference-service] sock = connection.create_connection( [e2e-llm-inference-service] (self._dns_host, self.port), [e2e-llm-inference-service] self.timeout, [e2e-llm-inference-service] source_address=self.source_address, [e2e-llm-inference-service] socket_options=self.socket_options, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except socket.gaierror as e: [e2e-llm-inference-service] > raise NameResolutionError(self.host, self, e) from e [e2e-llm-inference-service] E urllib3.exceptions.NameResolutionError: HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:211: NameResolutionError [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] request = , stream = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), verify = '/tmp/ca.crt' [e2e-llm-inference-service] cert = None, proxies = OrderedDict() [e2e-llm-inference-service] [e2e-llm-inference-service] def send( [e2e-llm-inference-service] self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Sends PreparedRequest object. Returns Response object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param request: The :class:`PreparedRequest ` being sent. [e2e-llm-inference-service] :param stream: (optional) Whether to stream the request content. [e2e-llm-inference-service] :param timeout: (optional) How long to wait for the server to send [e2e-llm-inference-service] data before giving up, as a float, or a :ref:`(connect timeout, [e2e-llm-inference-service] read timeout) ` tuple. [e2e-llm-inference-service] :type timeout: float or tuple or urllib3 Timeout object [e2e-llm-inference-service] :param verify: (optional) Either a boolean, in which case it controls whether [e2e-llm-inference-service] we verify the server's TLS certificate, or a string, in which case it [e2e-llm-inference-service] must be a path to a CA bundle to use [e2e-llm-inference-service] :param cert: (optional) Any user-provided SSL certificate to be trusted. [e2e-llm-inference-service] :param proxies: (optional) The proxies dictionary to apply to the request. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] conn = self.get_connection_with_tls_context( [e2e-llm-inference-service] request, verify, proxies=proxies, cert=cert [e2e-llm-inference-service] ) [e2e-llm-inference-service] except LocationValueError as e: [e2e-llm-inference-service] raise InvalidURL(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] self.cert_verify(conn, request.url, verify, cert) [e2e-llm-inference-service] url = self.request_url(request, proxies) [e2e-llm-inference-service] self.add_headers( [e2e-llm-inference-service] request, [e2e-llm-inference-service] stream=stream, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] verify=verify, [e2e-llm-inference-service] cert=cert, [e2e-llm-inference-service] proxies=proxies, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] chunked = not (request.body is None or "Content-Length" in request.headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(timeout, tuple): [e2e-llm-inference-service] try: [e2e-llm-inference-service] connect, read = timeout [e2e-llm-inference-service] timeout = TimeoutSauce(connect=connect, read=read) [e2e-llm-inference-service] except ValueError: [e2e-llm-inference-service] raise ValueError( [e2e-llm-inference-service] f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, " [e2e-llm-inference-service] f"or a single float to set both timeouts to the same value." [e2e-llm-inference-service] ) [e2e-llm-inference-service] elif isinstance(timeout, TimeoutSauce): [e2e-llm-inference-service] pass [e2e-llm-inference-service] else: [e2e-llm-inference-service] timeout = TimeoutSauce(connect=timeout, read=timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > resp = conn.urlopen( [e2e-llm-inference-service] method=request.method, [e2e-llm-inference-service] url=url, [e2e-llm-inference-service] body=request.body, [e2e-llm-inference-service] headers=request.headers, [e2e-llm-inference-service] redirect=False, [e2e-llm-inference-service] assert_same_host=False, [e2e-llm-inference-service] preload_content=False, [e2e-llm-inference-service] decode_content=False, [e2e-llm-inference-service] retries=self.max_retries, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/adapters.py:667: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=7, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=6, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=5, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=4, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=3, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=2, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=1, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False, err = None, clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] > retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:841: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] response = None [e2e-llm-inference-service] error = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] _pool = [e2e-llm-inference-service] _stacktrace = [e2e-llm-inference-service] [e2e-llm-inference-service] def increment( [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str | None = None, [e2e-llm-inference-service] url: str | None = None, [e2e-llm-inference-service] response: BaseHTTPResponse | None = None, [e2e-llm-inference-service] error: Exception | None = None, [e2e-llm-inference-service] _pool: ConnectionPool | None = None, [e2e-llm-inference-service] _stacktrace: TracebackType | None = None, [e2e-llm-inference-service] ) -> Self: [e2e-llm-inference-service] """Return a new Retry object with incremented retry counters. [e2e-llm-inference-service] [e2e-llm-inference-service] :param response: A response object, or None, if the server did not [e2e-llm-inference-service] return a response. [e2e-llm-inference-service] :type response: :class:`~urllib3.response.BaseHTTPResponse` [e2e-llm-inference-service] :param Exception error: An error encountered during the request, or [e2e-llm-inference-service] None if the response was received successfully. [e2e-llm-inference-service] [e2e-llm-inference-service] :return: A new ``Retry`` object. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if self.total is False and error: [e2e-llm-inference-service] # Disabled, indicate to re-raise the error. [e2e-llm-inference-service] raise reraise(type(error), error, _stacktrace) [e2e-llm-inference-service] [e2e-llm-inference-service] total = self.total [e2e-llm-inference-service] if total is not None: [e2e-llm-inference-service] total -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] connect = self.connect [e2e-llm-inference-service] read = self.read [e2e-llm-inference-service] redirect = self.redirect [e2e-llm-inference-service] status_count = self.status [e2e-llm-inference-service] other = self.other [e2e-llm-inference-service] cause = "unknown" [e2e-llm-inference-service] status = None [e2e-llm-inference-service] redirect_location = None [e2e-llm-inference-service] [e2e-llm-inference-service] if error and self._is_connection_error(error): [e2e-llm-inference-service] # Connect retry? [e2e-llm-inference-service] if connect is False: [e2e-llm-inference-service] raise reraise(type(error), error, _stacktrace) [e2e-llm-inference-service] elif connect is not None: [e2e-llm-inference-service] connect -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] elif error and self._is_read_error(error): [e2e-llm-inference-service] # Read retry? [e2e-llm-inference-service] if read is False or method is None or not self._is_method_retryable(method): [e2e-llm-inference-service] raise reraise(type(error), error, _stacktrace) [e2e-llm-inference-service] elif read is not None: [e2e-llm-inference-service] read -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] elif error: [e2e-llm-inference-service] # Other retry? [e2e-llm-inference-service] if other is not None: [e2e-llm-inference-service] other -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] elif response and response.get_redirect_location(): [e2e-llm-inference-service] # Redirect retry? [e2e-llm-inference-service] if redirect is not None: [e2e-llm-inference-service] redirect -= 1 [e2e-llm-inference-service] cause = "too many redirects" [e2e-llm-inference-service] response_redirect_location = response.get_redirect_location() [e2e-llm-inference-service] if response_redirect_location: [e2e-llm-inference-service] redirect_location = response_redirect_location [e2e-llm-inference-service] status = response.status [e2e-llm-inference-service] [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Incrementing because of a server error like a 500 in [e2e-llm-inference-service] # status_forcelist and the given method is in the allowed_methods [e2e-llm-inference-service] cause = ResponseError.GENERIC_ERROR [e2e-llm-inference-service] if response and response.status: [e2e-llm-inference-service] if status_count is not None: [e2e-llm-inference-service] status_count -= 1 [e2e-llm-inference-service] cause = ResponseError.SPECIFIC_ERROR.format(status_code=response.status) [e2e-llm-inference-service] status = response.status [e2e-llm-inference-service] [e2e-llm-inference-service] history = self.history + ( [e2e-llm-inference-service] RequestHistory(method, url, error, status, redirect_location), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] new_retry = self.new( [e2e-llm-inference-service] total=total, [e2e-llm-inference-service] connect=connect, [e2e-llm-inference-service] read=read, [e2e-llm-inference-service] redirect=redirect, [e2e-llm-inference-service] status=status_count, [e2e-llm-inference-service] other=other, [e2e-llm-inference-service] history=history, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if new_retry.is_exhausted(): [e2e-llm-inference-service] reason = error or ResponseError(cause) [e2e-llm-inference-service] > raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] [e2e-llm-inference-service] E urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/retry.py:519: MaxRetryError [e2e-llm-inference-service] [e2e-llm-inference-service] During handling of the above exception, another exception occurred: [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_model_responds(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name, [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] > response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:520: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] url = 'http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] [e2e-llm-inference-service] def post_with_retry( [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] *, [e2e-llm-inference-service] headers: Dict = None, [e2e-llm-inference-service] json_data: Union[Dict, List] = None, [e2e-llm-inference-service] data: Union[str, bytes] = None, [e2e-llm-inference-service] stream: bool = False, [e2e-llm-inference-service] timeout: float = None, [e2e-llm-inference-service] total_retries: int = DEFAULT_RETRY_TOTAL, [e2e-llm-inference-service] backoff_factor: float = DEFAULT_RETRY_BACKOFF_FACTOR, [e2e-llm-inference-service] retry_status_codes=DEFAULT_RETRY_STATUS_CODES, [e2e-llm-inference-service] ) -> requests.Response: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Send POST request with retries for transient HTTP and network failures. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if json_data is not None and data is not None: [e2e-llm-inference-service] raise ValueError("Only one of json_data or data can be provided.") [e2e-llm-inference-service] [e2e-llm-inference-service] retry = Retry( [e2e-llm-inference-service] total=total_retries, [e2e-llm-inference-service] backoff_factor=backoff_factor, [e2e-llm-inference-service] status_forcelist=retry_status_codes, [e2e-llm-inference-service] allowed_methods=["POST"], [e2e-llm-inference-service] raise_on_status=False, [e2e-llm-inference-service] ) [e2e-llm-inference-service] with requests.Session() as session: [e2e-llm-inference-service] session.mount("http://", HTTPAdapter(max_retries=retry)) [e2e-llm-inference-service] session.mount("https://", HTTPAdapter(max_retries=retry)) [e2e-llm-inference-service] > return session.post( [e2e-llm-inference-service] url, [e2e-llm-inference-service] json=json_data, [e2e-llm-inference-service] data=data, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] stream=stream, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] common/http_retry.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] data = None [e2e-llm-inference-service] json = {'max_tokens': 100, 'model': 'facebook/opt-125m', 'prompt': 'KServe is a'} [e2e-llm-inference-service] kwargs = {'headers': {'Content-Type': 'application/json'}, 'stream': False, 'timeout': 60} [e2e-llm-inference-service] [e2e-llm-inference-service] def post(self, url, data=None, json=None, **kwargs): [e2e-llm-inference-service] r"""Sends a POST request. Returns :class:`Response` object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: URL for the new :class:`Request` object. [e2e-llm-inference-service] :param data: (optional) Dictionary, list of tuples, bytes, or file-like [e2e-llm-inference-service] object to send in the body of the :class:`Request`. [e2e-llm-inference-service] :param json: (optional) json to send in the body of the :class:`Request`. [e2e-llm-inference-service] :param \*\*kwargs: Optional arguments that ``request`` takes. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.request("POST", url, data=data, json=json, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/sessions.py:637: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = , method = 'POST' [e2e-llm-inference-service] url = 'http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions' [e2e-llm-inference-service] params = None, data = None, headers = {'Content-Type': 'application/json'} [e2e-llm-inference-service] cookies = None, files = None, auth = None, timeout = 60, allow_redirects = True [e2e-llm-inference-service] proxies = {}, hooks = None, stream = False, verify = None, cert = None [e2e-llm-inference-service] json = {'max_tokens': 100, 'model': 'facebook/opt-125m', 'prompt': 'KServe is a'} [e2e-llm-inference-service] [e2e-llm-inference-service] def request( [e2e-llm-inference-service] self, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] params=None, [e2e-llm-inference-service] data=None, [e2e-llm-inference-service] headers=None, [e2e-llm-inference-service] cookies=None, [e2e-llm-inference-service] files=None, [e2e-llm-inference-service] auth=None, [e2e-llm-inference-service] timeout=None, [e2e-llm-inference-service] allow_redirects=True, [e2e-llm-inference-service] proxies=None, [e2e-llm-inference-service] hooks=None, [e2e-llm-inference-service] stream=None, [e2e-llm-inference-service] verify=None, [e2e-llm-inference-service] cert=None, [e2e-llm-inference-service] json=None, [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Constructs a :class:`Request `, prepares it and sends it. [e2e-llm-inference-service] Returns :class:`Response ` object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: method for the new :class:`Request` object. [e2e-llm-inference-service] :param url: URL for the new :class:`Request` object. [e2e-llm-inference-service] :param params: (optional) Dictionary or bytes to be sent in the query [e2e-llm-inference-service] string for the :class:`Request`. [e2e-llm-inference-service] :param data: (optional) Dictionary, list of tuples, bytes, or file-like [e2e-llm-inference-service] object to send in the body of the :class:`Request`. [e2e-llm-inference-service] :param json: (optional) json to send in the body of the [e2e-llm-inference-service] :class:`Request`. [e2e-llm-inference-service] :param headers: (optional) Dictionary of HTTP Headers to send with the [e2e-llm-inference-service] :class:`Request`. [e2e-llm-inference-service] :param cookies: (optional) Dict or CookieJar object to send with the [e2e-llm-inference-service] :class:`Request`. [e2e-llm-inference-service] :param files: (optional) Dictionary of ``'filename': file-like-objects`` [e2e-llm-inference-service] for multipart encoding upload. [e2e-llm-inference-service] :param auth: (optional) Auth tuple or callable to enable [e2e-llm-inference-service] Basic/Digest/Custom HTTP Auth. [e2e-llm-inference-service] :param timeout: (optional) How long to wait for the server to send [e2e-llm-inference-service] data before giving up, as a float, or a :ref:`(connect timeout, [e2e-llm-inference-service] read timeout) ` tuple. [e2e-llm-inference-service] :type timeout: float or tuple [e2e-llm-inference-service] :param allow_redirects: (optional) Set to True by default. [e2e-llm-inference-service] :type allow_redirects: bool [e2e-llm-inference-service] :param proxies: (optional) Dictionary mapping protocol or protocol and [e2e-llm-inference-service] hostname to the URL of the proxy. [e2e-llm-inference-service] :param hooks: (optional) Dictionary mapping hook name to one event or [e2e-llm-inference-service] list of events, event must be callable. [e2e-llm-inference-service] :param stream: (optional) whether to immediately download the response [e2e-llm-inference-service] content. Defaults to ``False``. [e2e-llm-inference-service] :param verify: (optional) Either a boolean, in which case it controls whether we verify [e2e-llm-inference-service] the server's TLS certificate, or a string, in which case it must be a path [e2e-llm-inference-service] to a CA bundle to use. Defaults to ``True``. When set to [e2e-llm-inference-service] ``False``, requests will accept any TLS certificate presented by [e2e-llm-inference-service] the server, and will ignore hostname mismatches and/or expired [e2e-llm-inference-service] certificates, which will make your application vulnerable to [e2e-llm-inference-service] man-in-the-middle (MitM) attacks. Setting verify to ``False`` [e2e-llm-inference-service] may be useful during local development or testing. [e2e-llm-inference-service] :param cert: (optional) if String, path to ssl client cert file (.pem). [e2e-llm-inference-service] If Tuple, ('cert', 'key') pair. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] # Create the Request. [e2e-llm-inference-service] req = Request( [e2e-llm-inference-service] method=method.upper(), [e2e-llm-inference-service] url=url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] files=files, [e2e-llm-inference-service] data=data or {}, [e2e-llm-inference-service] json=json, [e2e-llm-inference-service] params=params or {}, [e2e-llm-inference-service] auth=auth, [e2e-llm-inference-service] cookies=cookies, [e2e-llm-inference-service] hooks=hooks, [e2e-llm-inference-service] ) [e2e-llm-inference-service] prep = self.prepare_request(req) [e2e-llm-inference-service] [e2e-llm-inference-service] proxies = proxies or {} [e2e-llm-inference-service] [e2e-llm-inference-service] settings = self.merge_environment_settings( [e2e-llm-inference-service] prep.url, proxies, stream, verify, cert [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Send the request. [e2e-llm-inference-service] send_kwargs = { [e2e-llm-inference-service] "timeout": timeout, [e2e-llm-inference-service] "allow_redirects": allow_redirects, [e2e-llm-inference-service] } [e2e-llm-inference-service] send_kwargs.update(settings) [e2e-llm-inference-service] > resp = self.send(prep, **send_kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/sessions.py:589: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] request = [e2e-llm-inference-service] kwargs = {'cert': None, 'proxies': OrderedDict(), 'stream': False, 'timeout': 60, ...} [e2e-llm-inference-service] allow_redirects = True, stream = False, hooks = {'response': []} [e2e-llm-inference-service] adapter = [e2e-llm-inference-service] start = 1776970372.049822 [e2e-llm-inference-service] [e2e-llm-inference-service] def send(self, request, **kwargs): [e2e-llm-inference-service] """Send a given PreparedRequest. [e2e-llm-inference-service] [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] # Set defaults that the hooks can utilize to ensure they always have [e2e-llm-inference-service] # the correct parameters to reproduce the previous request. [e2e-llm-inference-service] kwargs.setdefault("stream", self.stream) [e2e-llm-inference-service] kwargs.setdefault("verify", self.verify) [e2e-llm-inference-service] kwargs.setdefault("cert", self.cert) [e2e-llm-inference-service] if "proxies" not in kwargs: [e2e-llm-inference-service] kwargs["proxies"] = resolve_proxies(request, self.proxies, self.trust_env) [e2e-llm-inference-service] [e2e-llm-inference-service] # It's possible that users might accidentally send a Request object. [e2e-llm-inference-service] # Guard against that specific failure case. [e2e-llm-inference-service] if isinstance(request, Request): [e2e-llm-inference-service] raise ValueError("You can only send PreparedRequests.") [e2e-llm-inference-service] [e2e-llm-inference-service] # Set up variables needed for resolve_redirects and dispatching of hooks [e2e-llm-inference-service] allow_redirects = kwargs.pop("allow_redirects", True) [e2e-llm-inference-service] stream = kwargs.get("stream") [e2e-llm-inference-service] hooks = request.hooks [e2e-llm-inference-service] [e2e-llm-inference-service] # Get the appropriate adapter to use [e2e-llm-inference-service] adapter = self.get_adapter(url=request.url) [e2e-llm-inference-service] [e2e-llm-inference-service] # Start time (approximately) of the request [e2e-llm-inference-service] start = preferred_clock() [e2e-llm-inference-service] [e2e-llm-inference-service] # Send the request [e2e-llm-inference-service] > r = adapter.send(request, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/sessions.py:703: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] request = , stream = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), verify = '/tmp/ca.crt' [e2e-llm-inference-service] cert = None, proxies = OrderedDict() [e2e-llm-inference-service] [e2e-llm-inference-service] def send( [e2e-llm-inference-service] self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Sends PreparedRequest object. Returns Response object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param request: The :class:`PreparedRequest ` being sent. [e2e-llm-inference-service] :param stream: (optional) Whether to stream the request content. [e2e-llm-inference-service] :param timeout: (optional) How long to wait for the server to send [e2e-llm-inference-service] data before giving up, as a float, or a :ref:`(connect timeout, [e2e-llm-inference-service] read timeout) ` tuple. [e2e-llm-inference-service] :type timeout: float or tuple or urllib3 Timeout object [e2e-llm-inference-service] :param verify: (optional) Either a boolean, in which case it controls whether [e2e-llm-inference-service] we verify the server's TLS certificate, or a string, in which case it [e2e-llm-inference-service] must be a path to a CA bundle to use [e2e-llm-inference-service] :param cert: (optional) Any user-provided SSL certificate to be trusted. [e2e-llm-inference-service] :param proxies: (optional) The proxies dictionary to apply to the request. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] conn = self.get_connection_with_tls_context( [e2e-llm-inference-service] request, verify, proxies=proxies, cert=cert [e2e-llm-inference-service] ) [e2e-llm-inference-service] except LocationValueError as e: [e2e-llm-inference-service] raise InvalidURL(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] self.cert_verify(conn, request.url, verify, cert) [e2e-llm-inference-service] url = self.request_url(request, proxies) [e2e-llm-inference-service] self.add_headers( [e2e-llm-inference-service] request, [e2e-llm-inference-service] stream=stream, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] verify=verify, [e2e-llm-inference-service] cert=cert, [e2e-llm-inference-service] proxies=proxies, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] chunked = not (request.body is None or "Content-Length" in request.headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(timeout, tuple): [e2e-llm-inference-service] try: [e2e-llm-inference-service] connect, read = timeout [e2e-llm-inference-service] timeout = TimeoutSauce(connect=connect, read=read) [e2e-llm-inference-service] except ValueError: [e2e-llm-inference-service] raise ValueError( [e2e-llm-inference-service] f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, " [e2e-llm-inference-service] f"or a single float to set both timeouts to the same value." [e2e-llm-inference-service] ) [e2e-llm-inference-service] elif isinstance(timeout, TimeoutSauce): [e2e-llm-inference-service] pass [e2e-llm-inference-service] else: [e2e-llm-inference-service] timeout = TimeoutSauce(connect=timeout, read=timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] resp = conn.urlopen( [e2e-llm-inference-service] method=request.method, [e2e-llm-inference-service] url=url, [e2e-llm-inference-service] body=request.body, [e2e-llm-inference-service] headers=request.headers, [e2e-llm-inference-service] redirect=False, [e2e-llm-inference-service] assert_same_host=False, [e2e-llm-inference-service] preload_content=False, [e2e-llm-inference-service] decode_content=False, [e2e-llm-inference-service] retries=self.max_retries, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] except (ProtocolError, OSError) as err: [e2e-llm-inference-service] raise ConnectionError(err, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] except MaxRetryError as e: [e2e-llm-inference-service] if isinstance(e.reason, ConnectTimeoutError): [e2e-llm-inference-service] # TODO: Remove this in 3.0.0: see #2811 [e2e-llm-inference-service] if not isinstance(e.reason, NewConnectionError): [e2e-llm-inference-service] raise ConnectTimeout(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(e.reason, ResponseError): [e2e-llm-inference-service] raise RetryError(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(e.reason, _ProxyError): [e2e-llm-inference-service] raise ProxyError(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(e.reason, _SSLError): [e2e-llm-inference-service] # This branch is for urllib3 v1.22 and later. [e2e-llm-inference-service] raise SSLError(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] > raise ConnectionError(e, request=request) [e2e-llm-inference-service] E requests.exceptions.ConnectionError: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/adapters.py:700: ConnectionError [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name=... {'name': 'model-fb-opt-125m-llmisvc-model-2416c917'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] > wait_for_model_response(kserve_client, test_case, test_case.wait_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:413: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'workload-sin... {'name': 'model-fb-opt-125m-llmisvc-model-2416c917'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-04-23T18:40:33.691094', start_time = 1776969633.691368 [e2e-llm-inference-service] duration = 1102.4803366661072, timestamp_end = '2026-04-23T18:58:56.171708' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name=... {'name': 'model-fb-opt-125m-llmisvc-model-2416c917'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900, extra_headers = None [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_model_responds(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name, [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.response_assertion(response) [e2e-llm-inference-service] return response.text[: test_case.max_tokens] [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_model_responds, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:535: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_model_responds at 0x7fb9c4b8d800> [e2e-llm-inference-service] timeout = 900, interval = 5.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:628: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_model_responds(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name, [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] > raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] E AssertionError: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:528: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1107 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1056 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-fb-6a5fa48c in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1082 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-fb-6a5fa48c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1092 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-fb-6a5fa48c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1056 Checking LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-39729a12 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1082 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-39729a12 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1092 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-39729a12 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1056 Checking LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-2416c917 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1082 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-2416c917 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1092 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-2416c917 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-04-23T18:37:34.481339] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name='llmisvc-model-fb-opt-125m-route-55f7ae4a', endpoint='/v1/completions', max_tokens=100, payload_formatter=, response_assertion=, wait_timeout=900, response_timeout=60, before_test=[], after_test=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-55f7ae4a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-6a5fa48c'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-39729a12'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-2416c917'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1107 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-04-23T18:37:34.494114] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-55f7ae4a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-6a5fa48c'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-39729a12'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-2416c917'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-04-23T18:37:34.585350] end - ✅ in 0.091s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-04-23T18:37:34.585502] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-55f7ae4a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-6a5fa48c'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-39729a12'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-2416c917'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:34.585829] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:34.673802] end - ✅ in 0.088s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:35.674170] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:35.773844] end - ✅ in 0.099s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:36.774142] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:36.782303] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:37.782700] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:37.791128] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:38.791483] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:38.799486] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:39.799883] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:39.808418] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:40.808810] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:40.816644] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:41.817096] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:41.825182] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:42.825406] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:42.832574] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:43.832976] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:43.840635] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:44.841042] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:44.849416] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:45.849718] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:45.857327] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:46.857589] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:46.864979] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:47.865251] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:47.873822] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:48.874270] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:48.882445] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:49.882702] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:49.891713] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:50.892055] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:50.900086] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:51.900469] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:51.908640] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:52.908991] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:52.916742] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:53.917153] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:53.924740] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:54.925075] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:54.932683] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:55.932971] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:55.940845] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:56.941267] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:56.951101] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:57.951461] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:57.959127] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:58.959521] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:58.968032] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:37:59.968292] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:37:59.976299] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:00.976678] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:00.984642] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:01.985033] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:01.992856] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:02.993139] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:03.078620] end - ✅ in 0.085s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:04.078964] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:04.174770] end - ✅ in 0.096s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:05.175245] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:05.275033] end - ✅ in 0.100s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:06.275330] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:06.299527] end - ✅ in 0.024s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:07.299998] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:07.308983] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:08.309321] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:08.317787] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:09.318277] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:09.327612] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:10.328086] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:10.336723] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:11.337263] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:11.345724] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:12.346000] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:12.354299] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:13.354577] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:13.374097] end - ✅ in 0.019s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:14.374497] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:14.382664] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:15.382983] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:15.391432] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:16.391706] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:16.399758] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:17.400074] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:17.408989] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:18.409456] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:18.417446] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:19.417785] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:19.426096] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:20.426556] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:20.435293] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:21.435771] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:21.445246] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:22.445527] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:22.454404] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:23.454818] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:23.463539] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:24.463879] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:24.472163] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:25.472449] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:25.480866] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:26.481198] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:26.490762] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:27.491070] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:27.504320] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:28.504762] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:28.513934] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:29.514209] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:29.523617] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:30.523988] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:30.532835] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:31.533170] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:31.541366] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:32.541764] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:32.550067] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:33.550323] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:33.561135] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:34.561408] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:34.569977] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:35.570259] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:35.578779] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:36.579272] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:36.588119] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:37.588378] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:37.596924] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:38.597266] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:38.607081] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:39.607378] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:39.616034] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:40.616380] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:40.624895] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:41.625413] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:41.633962] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:42.634416] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:42.643271] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:43.643579] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:43.652369] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:44.652641] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:44.661392] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:45.661698] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:45.672432] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:46.672984] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:46.682161] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:47.682596] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:47.691633] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:48.691951] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:48.700450] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:49.700864] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:49.709127] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:50.709566] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:50.718139] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:51.718557] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:51.729330] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:52.729741] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:52.738526] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:53.738848] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:53.751293] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:54.751742] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:54.760365] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:55.760804] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:55.769321] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:56.769842] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:56.778661] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:57.778949] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:57.787408] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:58.787738] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:58.796766] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:38:59.797100] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:38:59.805929] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:00.806178] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:00.815350] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:01.815706] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:01.826188] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:02.826480] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:02.835806] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:03.836190] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:03.844369] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:04.844688] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:04.853612] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:05.853985] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:05.869875] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:06.870268] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:06.879038] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:07.879347] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:07.889206] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:08.889498] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:08.899478] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:09.899843] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:09.908257] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:10.908552] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:10.917218] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:11.917491] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:11.926219] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:12.926487] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:12.934682] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:13.935203] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:13.943655] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:14.944187] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:14.953130] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:15.953517] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:15.961895] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:16.962240] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:16.971455] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:17.971786] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:17.981429] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:18.981750] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:18.992177] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:19.992611] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:20.001081] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:21.001431] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:21.010675] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:22.010964] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:22.019722] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:23.019985] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:23.028346] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:24.028767] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:24.036675] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:25.037011] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:25.046286] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:26.046721] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:26.055438] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:27.055718] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:27.063954] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:28.064306] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:28.072681] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:29.073037] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:29.081702] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:30.082093] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:30.090849] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:31.091173] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:31.100793] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:32.101265] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:32.110429] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:33.110753] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:33.119984] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:34.120261] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:34.131290] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:35.131577] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:35.148530] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:36.148970] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:36.157767] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:37.158251] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:37.166969] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:38.167324] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:38.184795] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:39.185120] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:39.194044] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:40.194340] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:40.204251] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:41.204527] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:41.215357] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:42.215641] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:42.227561] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:43.227873] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:43.236296] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:44.236597] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:44.245402] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:45.245679] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:45.255950] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:46.256448] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:46.265492] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:47.265739] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:47.274104] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:48.274433] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:48.282597] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:49.282878] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:49.292609] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:50.292887] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:50.301303] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:51.301603] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:51.310841] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:52.311156] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:52.319698] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:53.319975] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:53.328556] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:54.328869] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:54.337615] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:55.337963] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:55.346697] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:56.346994] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:56.355810] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:57.356128] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:57.364555] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:58.364968] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:58.373766] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:39:59.374092] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:39:59.382957] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:00.383267] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:00.391866] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:01.392179] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:01.400732] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:02.401149] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:02.409985] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:03.410286] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:03.419320] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:04.419711] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:04.427560] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:05.427963] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:05.436650] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:06.437176] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:06.445762] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:07.446283] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:07.455156] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:08.455440] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:08.464146] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:09.464424] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:09.472588] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:10.472866] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:10.480635] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:11.480982] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:11.489674] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:12.490176] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:12.499118] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:13.499465] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:13.508783] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:14.509313] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:14.517796] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:15.518282] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:15.527075] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:16.527387] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:16.536430] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:17.536697] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:17.546666] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:18.547080] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:18.554811] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:19.555112] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:19.563027] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:20.563337] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:20.571307] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:21.571699] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:21.580418] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:22.580799] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:22.589366] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:23.589622] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:23.597197] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:24.597504] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:24.607028] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:25.607400] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:25.615988] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:26.616306] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:26.626674] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:27.626996] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:27.635890] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:28.636226] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:28.644848] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:29.645179] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:29.653993] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:30.654288] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:30.663008] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:31.663247] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:31.671496] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:32.671805] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:32.681301] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:38:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:38:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:38:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:33.681594] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:33.690844] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-04-23T18:40:33.691042] end - ✅ in 179.105s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-04-23T18:40:33.691094] start - args=(, TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name='llmisvc-model-fb-opt-125m-route-55f7ae4a', endpoint='/v1/completions', max_tokens=100, payload_formatter=, response_assertion=, wait_timeout=900, response_timeout=60, before_test=[], after_test=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-55f7ae4a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-6a5fa48c'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-39729a12'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-2416c917'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-04-23T18:40:33.691480] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-55f7ae4a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-6a5fa48c'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-39729a12'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-2416c917'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:33.692218] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:33.699763] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-04-23T18:40:33.699978] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:518 Calling LLM service at http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 100} [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] ERROR e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:527 ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-04-23T18:46:42.835371] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-55f7ae4a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-6a5fa48c'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-39729a12'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-2416c917'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:46:42.835970] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:46:42.883454] end - ✅ in 0.047s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-04-23T18:46:42.883651] end - ✅ in 0.048s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:518 Calling LLM service at http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 100} [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] ERROR e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:527 ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-04-23T18:52:51.994953] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-55f7ae4a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-6a5fa48c'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-39729a12'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-2416c917'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:52:51.995430] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:52:52.048699] end - ✅ in 0.053s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-04-23T18:52:52.048876] end - ✅ in 0.053s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:518 Calling LLM service at http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 100} [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions [e2e-llm-inference-service] ERROR e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:527 ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-04-23T18:58:56.171708] end - ❌ 1102.480s: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:58:56.171804] start - args=(, 'llmisvc-model-fb-opt-125m-route-55f7ae4a', 'kserve-ci-e2e-test'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:58:56.212151] end - ✅ in 0.040s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-04-23T18:58:57.469197] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-55f7ae4a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-6a5fa48c'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-39729a12'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-2416c917'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-04-23T18:58:57.502473] end - ✅ in 0.033s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-04-23T18:58:57.502556] end - ❌ 1283.021s: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] [e2e-llm-inference-service] def _new_conn(self) -> socket.socket: [e2e-llm-inference-service] """Establish a socket connection and set nodelay settings on it. [e2e-llm-inference-service] [e2e-llm-inference-service] :return: New socket connection. [e2e-llm-inference-service] """ [e2e-llm-inference-service] try: [e2e-llm-inference-service] > sock = connection.create_connection( [e2e-llm-inference-service] (self._dns_host, self.port), [e2e-llm-inference-service] self.timeout, [e2e-llm-inference-service] source_address=self.source_address, [e2e-llm-inference-service] socket_options=self.socket_options, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] address = ('openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', 80) [e2e-llm-inference-service] timeout = 60, source_address = None, socket_options = [(6, 1, 1)] [e2e-llm-inference-service] [e2e-llm-inference-service] def create_connection( [e2e-llm-inference-service] address: tuple[str, int], [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] source_address: tuple[str, int] | None = None, [e2e-llm-inference-service] socket_options: _TYPE_SOCKET_OPTIONS | None = None, [e2e-llm-inference-service] ) -> socket.socket: [e2e-llm-inference-service] """Connect to *address* and return the socket object. [e2e-llm-inference-service] [e2e-llm-inference-service] Convenience function. Connect to *address* (a 2-tuple ``(host, [e2e-llm-inference-service] port)``) and return the socket object. Passing the optional [e2e-llm-inference-service] *timeout* parameter will set the timeout on the socket instance [e2e-llm-inference-service] before attempting to connect. If no *timeout* is supplied, the [e2e-llm-inference-service] global default timeout setting returned by :func:`socket.getdefaulttimeout` [e2e-llm-inference-service] is used. If *source_address* is set it must be a tuple of (host, port) [e2e-llm-inference-service] for the socket to bind as a source address before making the connection. [e2e-llm-inference-service] An host of '' or port 0 tells the OS to use the default. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] host, port = address [e2e-llm-inference-service] if host.startswith("["): [e2e-llm-inference-service] host = host.strip("[]") [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Using the value from allowed_gai_family() in the context of getaddrinfo lets [e2e-llm-inference-service] # us select whether to work with IPv4 DNS records, IPv6 records, or both. [e2e-llm-inference-service] # The original create_connection function always returns all records. [e2e-llm-inference-service] family = allowed_gai_family() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] host.encode("idna") [e2e-llm-inference-service] except UnicodeError: [e2e-llm-inference-service] raise LocationParseError(f"'{host}', label empty or too long") from None [e2e-llm-inference-service] [e2e-llm-inference-service] > for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/connection.py:60: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] host = 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' [e2e-llm-inference-service] port = 80, family = [e2e-llm-inference-service] type = , proto = 0, flags = 0 [e2e-llm-inference-service] [e2e-llm-inference-service] def getaddrinfo(host, port, family=0, type=0, proto=0, flags=0): [e2e-llm-inference-service] """Resolve host and port into list of address info entries. [e2e-llm-inference-service] [e2e-llm-inference-service] Translate the host/port argument into a sequence of 5-tuples that contain [e2e-llm-inference-service] all the necessary arguments for creating a socket connected to that service. [e2e-llm-inference-service] host is a domain name, a string representation of an IPv4/v6 address or [e2e-llm-inference-service] None. port is a string service name such as 'http', a numeric port number or [e2e-llm-inference-service] None. By passing None as the value of host and port, you can pass NULL to [e2e-llm-inference-service] the underlying C API. [e2e-llm-inference-service] [e2e-llm-inference-service] The family, type and proto arguments can be optionally specified in order to [e2e-llm-inference-service] narrow the list of addresses returned. Passing zero as a value for each of [e2e-llm-inference-service] these arguments selects the full range of results. [e2e-llm-inference-service] """ [e2e-llm-inference-service] # We override this function since we want to translate the numeric family [e2e-llm-inference-service] # and socket type values to enum constants. [e2e-llm-inference-service] addrlist = [] [e2e-llm-inference-service] > for res in _socket.getaddrinfo(host, port, family, type, proto, flags): [e2e-llm-inference-service] E socket.gaierror: [Errno -2] Name or service not known [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/socket.py:974: gaierror [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False, err = None, clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] > response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:787: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] conn = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), chunked = False [e2e-llm-inference-service] response_conn = [e2e-llm-inference-service] preload_content = False, decode_content = False, enforce_content_length = True [e2e-llm-inference-service] [e2e-llm-inference-service] def _make_request( [e2e-llm-inference-service] self, [e2e-llm-inference-service] conn: BaseHTTPConnection, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | None = None, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] response_conn: BaseHTTPConnection | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] enforce_content_length: bool = True, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Perform a request on a given urllib connection object taken from our [e2e-llm-inference-service] pool. [e2e-llm-inference-service] [e2e-llm-inference-service] :param conn: [e2e-llm-inference-service] a connection from one of our connection pools [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] Pass ``None`` to retry until you receive a response. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param response_conn: [e2e-llm-inference-service] Set this to ``None`` if you will handle releasing the connection or [e2e-llm-inference-service] set the connection to have the response release it. [e2e-llm-inference-service] [e2e-llm-inference-service] :param preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded during construction. [e2e-llm-inference-service] [e2e-llm-inference-service] :param decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param enforce_content_length: [e2e-llm-inference-service] Enforce content length checking. Body returned by server must match [e2e-llm-inference-service] value of Content-Length header, if present. Otherwise, raise error. [e2e-llm-inference-service] """ [e2e-llm-inference-service] self.num_requests += 1 [e2e-llm-inference-service] [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] timeout_obj.start_connect() [e2e-llm-inference-service] conn.timeout = Timeout.resolve_default_timeout(timeout_obj.connect_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Trigger any extra validation we need to do. [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._validate_conn(conn) [e2e-llm-inference-service] except (SocketTimeout, BaseSSLError) as e: [e2e-llm-inference-service] self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # _validate_conn() starts the connection to an HTTPS proxy [e2e-llm-inference-service] # so we need to wrap errors with 'ProxyError' here too. [e2e-llm-inference-service] except ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] # If the connection didn't successfully connect to it's proxy [e2e-llm-inference-service] # then there [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, (OSError, NewConnectionError, TimeoutError, SSLError) [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] raise new_e [e2e-llm-inference-service] [e2e-llm-inference-service] # conn.request() calls http.client.*.request, not the method in [e2e-llm-inference-service] # urllib3.request. It also calls makefile (recv) on the socket. [e2e-llm-inference-service] try: [e2e-llm-inference-service] > conn.request( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] enforce_content_length=enforce_content_length, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:493: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] [e2e-llm-inference-service] def request( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] *, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] enforce_content_length: bool = True, [e2e-llm-inference-service] ) -> None: [e2e-llm-inference-service] # Update the inner socket's timeout value to send the request. [e2e-llm-inference-service] # This only triggers if the connection is re-used. [e2e-llm-inference-service] if self.sock is not None: [e2e-llm-inference-service] self.sock.settimeout(self.timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] # Store these values to be fed into the HTTPResponse [e2e-llm-inference-service] # object later. TODO: Remove this in favor of a real [e2e-llm-inference-service] # HTTP lifecycle mechanism. [e2e-llm-inference-service] [e2e-llm-inference-service] # We have to store these before we call .request() [e2e-llm-inference-service] # because sometimes we can still salvage a response [e2e-llm-inference-service] # off the wire even if we aren't able to completely [e2e-llm-inference-service] # send the request body. [e2e-llm-inference-service] self._response_options = _ResponseOptions( [e2e-llm-inference-service] request_method=method, [e2e-llm-inference-service] request_url=url, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] enforce_content_length=enforce_content_length, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = {} [e2e-llm-inference-service] header_keys = frozenset(to_str(k.lower()) for k in headers) [e2e-llm-inference-service] skip_accept_encoding = "accept-encoding" in header_keys [e2e-llm-inference-service] skip_host = "host" in header_keys [e2e-llm-inference-service] self.putrequest( [e2e-llm-inference-service] method, url, skip_accept_encoding=skip_accept_encoding, skip_host=skip_host [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Transform the body into an iterable of sendall()-able chunks [e2e-llm-inference-service] # and detect if an explicit Content-Length is doable. [e2e-llm-inference-service] chunks_and_cl = body_to_chunks(body, method=method, blocksize=self.blocksize) [e2e-llm-inference-service] chunks = chunks_and_cl.chunks [e2e-llm-inference-service] content_length = chunks_and_cl.content_length [e2e-llm-inference-service] [e2e-llm-inference-service] # When chunked is explicit set to 'True' we respect that. [e2e-llm-inference-service] if chunked: [e2e-llm-inference-service] if "transfer-encoding" not in header_keys: [e2e-llm-inference-service] self.putheader("Transfer-Encoding", "chunked") [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Detect whether a framing mechanism is already in use. If so [e2e-llm-inference-service] # we respect that value, otherwise we pick chunked vs content-length [e2e-llm-inference-service] # depending on the type of 'body'. [e2e-llm-inference-service] if "content-length" in header_keys: [e2e-llm-inference-service] chunked = False [e2e-llm-inference-service] elif "transfer-encoding" in header_keys: [e2e-llm-inference-service] chunked = True [e2e-llm-inference-service] [e2e-llm-inference-service] # Otherwise we go off the recommendation of 'body_to_chunks()'. [e2e-llm-inference-service] else: [e2e-llm-inference-service] chunked = False [e2e-llm-inference-service] if content_length is None: [e2e-llm-inference-service] if chunks is not None: [e2e-llm-inference-service] chunked = True [e2e-llm-inference-service] self.putheader("Transfer-Encoding", "chunked") [e2e-llm-inference-service] else: [e2e-llm-inference-service] self.putheader("Content-Length", str(content_length)) [e2e-llm-inference-service] [e2e-llm-inference-service] # Now that framing headers are out of the way we send all the other headers. [e2e-llm-inference-service] if "user-agent" not in header_keys: [e2e-llm-inference-service] self.putheader("User-Agent", _get_default_user_agent()) [e2e-llm-inference-service] for header, value in headers.items(): [e2e-llm-inference-service] self.putheader(header, value) [e2e-llm-inference-service] > self.endheaders() [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:500: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] message_body = None [e2e-llm-inference-service] [e2e-llm-inference-service] def endheaders(self, message_body=None, *, encode_chunked=False): [e2e-llm-inference-service] """Indicate that the last header line has been sent to the server. [e2e-llm-inference-service] [e2e-llm-inference-service] This method sends the request to the server. The optional message_body [e2e-llm-inference-service] argument can be used to pass a message body associated with the [e2e-llm-inference-service] request. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if self.__state == _CS_REQ_STARTED: [e2e-llm-inference-service] self.__state = _CS_REQ_SENT [e2e-llm-inference-service] else: [e2e-llm-inference-service] raise CannotSendHeader() [e2e-llm-inference-service] > self._send_output(message_body, encode_chunked=encode_chunked) [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/http/client.py:1298: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] message_body = None, encode_chunked = False [e2e-llm-inference-service] [e2e-llm-inference-service] def _send_output(self, message_body=None, encode_chunked=False): [e2e-llm-inference-service] """Send the currently buffered request and clear the buffer. [e2e-llm-inference-service] [e2e-llm-inference-service] Appends an extra \\r\\n to the buffer. [e2e-llm-inference-service] A message_body may be specified, to be appended to the request. [e2e-llm-inference-service] """ [e2e-llm-inference-service] self._buffer.extend((b"", b"")) [e2e-llm-inference-service] msg = b"\r\n".join(self._buffer) [e2e-llm-inference-service] del self._buffer[:] [e2e-llm-inference-service] > self.send(msg) [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/http/client.py:1058: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] data = b'POST /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions HTTP/1.1\r\nHost: openshift-ai-inference-opensh...gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Type: application/json\r\nContent-Length: 74\r\n\r\n' [e2e-llm-inference-service] [e2e-llm-inference-service] def send(self, data): [e2e-llm-inference-service] """Send `data' to the server. [e2e-llm-inference-service] ``data`` can be a string object, a bytes object, an array object, a [e2e-llm-inference-service] file-like object that supports a .read() method, or an iterable object. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] if self.sock is None: [e2e-llm-inference-service] if self.auto_open: [e2e-llm-inference-service] > self.connect() [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/http/client.py:996: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] [e2e-llm-inference-service] def connect(self) -> None: [e2e-llm-inference-service] > self.sock = self._new_conn() [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:331: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] [e2e-llm-inference-service] def _new_conn(self) -> socket.socket: [e2e-llm-inference-service] """Establish a socket connection and set nodelay settings on it. [e2e-llm-inference-service] [e2e-llm-inference-service] :return: New socket connection. [e2e-llm-inference-service] """ [e2e-llm-inference-service] try: [e2e-llm-inference-service] sock = connection.create_connection( [e2e-llm-inference-service] (self._dns_host, self.port), [e2e-llm-inference-service] self.timeout, [e2e-llm-inference-service] source_address=self.source_address, [e2e-llm-inference-service] socket_options=self.socket_options, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except socket.gaierror as e: [e2e-llm-inference-service] > raise NameResolutionError(self.host, self, e) from e [e2e-llm-inference-service] E urllib3.exceptions.NameResolutionError: HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:211: NameResolutionError [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] request = , stream = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), verify = '/tmp/ca.crt' [e2e-llm-inference-service] cert = None, proxies = OrderedDict() [e2e-llm-inference-service] [e2e-llm-inference-service] def send( [e2e-llm-inference-service] self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Sends PreparedRequest object. Returns Response object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param request: The :class:`PreparedRequest ` being sent. [e2e-llm-inference-service] :param stream: (optional) Whether to stream the request content. [e2e-llm-inference-service] :param timeout: (optional) How long to wait for the server to send [e2e-llm-inference-service] data before giving up, as a float, or a :ref:`(connect timeout, [e2e-llm-inference-service] read timeout) ` tuple. [e2e-llm-inference-service] :type timeout: float or tuple or urllib3 Timeout object [e2e-llm-inference-service] :param verify: (optional) Either a boolean, in which case it controls whether [e2e-llm-inference-service] we verify the server's TLS certificate, or a string, in which case it [e2e-llm-inference-service] must be a path to a CA bundle to use [e2e-llm-inference-service] :param cert: (optional) Any user-provided SSL certificate to be trusted. [e2e-llm-inference-service] :param proxies: (optional) The proxies dictionary to apply to the request. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] conn = self.get_connection_with_tls_context( [e2e-llm-inference-service] request, verify, proxies=proxies, cert=cert [e2e-llm-inference-service] ) [e2e-llm-inference-service] except LocationValueError as e: [e2e-llm-inference-service] raise InvalidURL(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] self.cert_verify(conn, request.url, verify, cert) [e2e-llm-inference-service] url = self.request_url(request, proxies) [e2e-llm-inference-service] self.add_headers( [e2e-llm-inference-service] request, [e2e-llm-inference-service] stream=stream, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] verify=verify, [e2e-llm-inference-service] cert=cert, [e2e-llm-inference-service] proxies=proxies, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] chunked = not (request.body is None or "Content-Length" in request.headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(timeout, tuple): [e2e-llm-inference-service] try: [e2e-llm-inference-service] connect, read = timeout [e2e-llm-inference-service] timeout = TimeoutSauce(connect=connect, read=read) [e2e-llm-inference-service] except ValueError: [e2e-llm-inference-service] raise ValueError( [e2e-llm-inference-service] f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, " [e2e-llm-inference-service] f"or a single float to set both timeouts to the same value." [e2e-llm-inference-service] ) [e2e-llm-inference-service] elif isinstance(timeout, TimeoutSauce): [e2e-llm-inference-service] pass [e2e-llm-inference-service] else: [e2e-llm-inference-service] timeout = TimeoutSauce(connect=timeout, read=timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > resp = conn.urlopen( [e2e-llm-inference-service] method=request.method, [e2e-llm-inference-service] url=url, [e2e-llm-inference-service] body=request.body, [e2e-llm-inference-service] headers=request.headers, [e2e-llm-inference-service] redirect=False, [e2e-llm-inference-service] assert_same_host=False, [e2e-llm-inference-service] preload_content=False, [e2e-llm-inference-service] decode_content=False, [e2e-llm-inference-service] retries=self.max_retries, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/adapters.py:667: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=7, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=6, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=5, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=4, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=3, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=2, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=1, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "KServe is a", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Content-Length': '74'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False, err = None, clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] > retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:841: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] response = None [e2e-llm-inference-service] error = NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local...'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)") [e2e-llm-inference-service] _pool = [e2e-llm-inference-service] _stacktrace = [e2e-llm-inference-service] [e2e-llm-inference-service] def increment( [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str | None = None, [e2e-llm-inference-service] url: str | None = None, [e2e-llm-inference-service] response: BaseHTTPResponse | None = None, [e2e-llm-inference-service] error: Exception | None = None, [e2e-llm-inference-service] _pool: ConnectionPool | None = None, [e2e-llm-inference-service] _stacktrace: TracebackType | None = None, [e2e-llm-inference-service] ) -> Self: [e2e-llm-inference-service] """Return a new Retry object with incremented retry counters. [e2e-llm-inference-service] [e2e-llm-inference-service] :param response: A response object, or None, if the server did not [e2e-llm-inference-service] return a response. [e2e-llm-inference-service] :type response: :class:`~urllib3.response.BaseHTTPResponse` [e2e-llm-inference-service] :param Exception error: An error encountered during the request, or [e2e-llm-inference-service] None if the response was received successfully. [e2e-llm-inference-service] [e2e-llm-inference-service] :return: A new ``Retry`` object. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if self.total is False and error: [e2e-llm-inference-service] # Disabled, indicate to re-raise the error. [e2e-llm-inference-service] raise reraise(type(error), error, _stacktrace) [e2e-llm-inference-service] [e2e-llm-inference-service] total = self.total [e2e-llm-inference-service] if total is not None: [e2e-llm-inference-service] total -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] connect = self.connect [e2e-llm-inference-service] read = self.read [e2e-llm-inference-service] redirect = self.redirect [e2e-llm-inference-service] status_count = self.status [e2e-llm-inference-service] other = self.other [e2e-llm-inference-service] cause = "unknown" [e2e-llm-inference-service] status = None [e2e-llm-inference-service] redirect_location = None [e2e-llm-inference-service] [e2e-llm-inference-service] if error and self._is_connection_error(error): [e2e-llm-inference-service] # Connect retry? [e2e-llm-inference-service] if connect is False: [e2e-llm-inference-service] raise reraise(type(error), error, _stacktrace) [e2e-llm-inference-service] elif connect is not None: [e2e-llm-inference-service] connect -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] elif error and self._is_read_error(error): [e2e-llm-inference-service] # Read retry? [e2e-llm-inference-service] if read is False or method is None or not self._is_method_retryable(method): [e2e-llm-inference-service] raise reraise(type(error), error, _stacktrace) [e2e-llm-inference-service] elif read is not None: [e2e-llm-inference-service] read -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] elif error: [e2e-llm-inference-service] # Other retry? [e2e-llm-inference-service] if other is not None: [e2e-llm-inference-service] other -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] elif response and response.get_redirect_location(): [e2e-llm-inference-service] # Redirect retry? [e2e-llm-inference-service] if redirect is not None: [e2e-llm-inference-service] redirect -= 1 [e2e-llm-inference-service] cause = "too many redirects" [e2e-llm-inference-service] response_redirect_location = response.get_redirect_location() [e2e-llm-inference-service] if response_redirect_location: [e2e-llm-inference-service] redirect_location = response_redirect_location [e2e-llm-inference-service] status = response.status [e2e-llm-inference-service] [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Incrementing because of a server error like a 500 in [e2e-llm-inference-service] # status_forcelist and the given method is in the allowed_methods [e2e-llm-inference-service] cause = ResponseError.GENERIC_ERROR [e2e-llm-inference-service] if response and response.status: [e2e-llm-inference-service] if status_count is not None: [e2e-llm-inference-service] status_count -= 1 [e2e-llm-inference-service] cause = ResponseError.SPECIFIC_ERROR.format(status_code=response.status) [e2e-llm-inference-service] status = response.status [e2e-llm-inference-service] [e2e-llm-inference-service] history = self.history + ( [e2e-llm-inference-service] RequestHistory(method, url, error, status, redirect_location), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] new_retry = self.new( [e2e-llm-inference-service] total=total, [e2e-llm-inference-service] connect=connect, [e2e-llm-inference-service] read=read, [e2e-llm-inference-service] redirect=redirect, [e2e-llm-inference-service] status=status_count, [e2e-llm-inference-service] other=other, [e2e-llm-inference-service] history=history, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if new_retry.is_exhausted(): [e2e-llm-inference-service] reason = error or ResponseError(cause) [e2e-llm-inference-service] > raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] [e2e-llm-inference-service] E urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/retry.py:519: MaxRetryError [e2e-llm-inference-service] [e2e-llm-inference-service] During handling of the above exception, another exception occurred: [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_model_responds(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name, [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] > response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:520: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] url = 'http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] [e2e-llm-inference-service] def post_with_retry( [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] *, [e2e-llm-inference-service] headers: Dict = None, [e2e-llm-inference-service] json_data: Union[Dict, List] = None, [e2e-llm-inference-service] data: Union[str, bytes] = None, [e2e-llm-inference-service] stream: bool = False, [e2e-llm-inference-service] timeout: float = None, [e2e-llm-inference-service] total_retries: int = DEFAULT_RETRY_TOTAL, [e2e-llm-inference-service] backoff_factor: float = DEFAULT_RETRY_BACKOFF_FACTOR, [e2e-llm-inference-service] retry_status_codes=DEFAULT_RETRY_STATUS_CODES, [e2e-llm-inference-service] ) -> requests.Response: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Send POST request with retries for transient HTTP and network failures. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if json_data is not None and data is not None: [e2e-llm-inference-service] raise ValueError("Only one of json_data or data can be provided.") [e2e-llm-inference-service] [e2e-llm-inference-service] retry = Retry( [e2e-llm-inference-service] total=total_retries, [e2e-llm-inference-service] backoff_factor=backoff_factor, [e2e-llm-inference-service] status_forcelist=retry_status_codes, [e2e-llm-inference-service] allowed_methods=["POST"], [e2e-llm-inference-service] raise_on_status=False, [e2e-llm-inference-service] ) [e2e-llm-inference-service] with requests.Session() as session: [e2e-llm-inference-service] session.mount("http://", HTTPAdapter(max_retries=retry)) [e2e-llm-inference-service] session.mount("https://", HTTPAdapter(max_retries=retry)) [e2e-llm-inference-service] > return session.post( [e2e-llm-inference-service] url, [e2e-llm-inference-service] json=json_data, [e2e-llm-inference-service] data=data, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] stream=stream, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] common/http_retry.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] data = None [e2e-llm-inference-service] json = {'max_tokens': 100, 'model': 'facebook/opt-125m', 'prompt': 'KServe is a'} [e2e-llm-inference-service] kwargs = {'headers': {'Content-Type': 'application/json'}, 'stream': False, 'timeout': 60} [e2e-llm-inference-service] [e2e-llm-inference-service] def post(self, url, data=None, json=None, **kwargs): [e2e-llm-inference-service] r"""Sends a POST request. Returns :class:`Response` object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: URL for the new :class:`Request` object. [e2e-llm-inference-service] :param data: (optional) Dictionary, list of tuples, bytes, or file-like [e2e-llm-inference-service] object to send in the body of the :class:`Request`. [e2e-llm-inference-service] :param json: (optional) json to send in the body of the :class:`Request`. [e2e-llm-inference-service] :param \*\*kwargs: Optional arguments that ``request`` takes. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.request("POST", url, data=data, json=json, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/sessions.py:637: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = , method = 'POST' [e2e-llm-inference-service] url = 'http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions' [e2e-llm-inference-service] params = None, data = None, headers = {'Content-Type': 'application/json'} [e2e-llm-inference-service] cookies = None, files = None, auth = None, timeout = 60, allow_redirects = True [e2e-llm-inference-service] proxies = {}, hooks = None, stream = False, verify = None, cert = None [e2e-llm-inference-service] json = {'max_tokens': 100, 'model': 'facebook/opt-125m', 'prompt': 'KServe is a'} [e2e-llm-inference-service] [e2e-llm-inference-service] def request( [e2e-llm-inference-service] self, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] params=None, [e2e-llm-inference-service] data=None, [e2e-llm-inference-service] headers=None, [e2e-llm-inference-service] cookies=None, [e2e-llm-inference-service] files=None, [e2e-llm-inference-service] auth=None, [e2e-llm-inference-service] timeout=None, [e2e-llm-inference-service] allow_redirects=True, [e2e-llm-inference-service] proxies=None, [e2e-llm-inference-service] hooks=None, [e2e-llm-inference-service] stream=None, [e2e-llm-inference-service] verify=None, [e2e-llm-inference-service] cert=None, [e2e-llm-inference-service] json=None, [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Constructs a :class:`Request `, prepares it and sends it. [e2e-llm-inference-service] Returns :class:`Response ` object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: method for the new :class:`Request` object. [e2e-llm-inference-service] :param url: URL for the new :class:`Request` object. [e2e-llm-inference-service] :param params: (optional) Dictionary or bytes to be sent in the query [e2e-llm-inference-service] string for the :class:`Request`. [e2e-llm-inference-service] :param data: (optional) Dictionary, list of tuples, bytes, or file-like [e2e-llm-inference-service] object to send in the body of the :class:`Request`. [e2e-llm-inference-service] :param json: (optional) json to send in the body of the [e2e-llm-inference-service] :class:`Request`. [e2e-llm-inference-service] :param headers: (optional) Dictionary of HTTP Headers to send with the [e2e-llm-inference-service] :class:`Request`. [e2e-llm-inference-service] :param cookies: (optional) Dict or CookieJar object to send with the [e2e-llm-inference-service] :class:`Request`. [e2e-llm-inference-service] :param files: (optional) Dictionary of ``'filename': file-like-objects`` [e2e-llm-inference-service] for multipart encoding upload. [e2e-llm-inference-service] :param auth: (optional) Auth tuple or callable to enable [e2e-llm-inference-service] Basic/Digest/Custom HTTP Auth. [e2e-llm-inference-service] :param timeout: (optional) How long to wait for the server to send [e2e-llm-inference-service] data before giving up, as a float, or a :ref:`(connect timeout, [e2e-llm-inference-service] read timeout) ` tuple. [e2e-llm-inference-service] :type timeout: float or tuple [e2e-llm-inference-service] :param allow_redirects: (optional) Set to True by default. [e2e-llm-inference-service] :type allow_redirects: bool [e2e-llm-inference-service] :param proxies: (optional) Dictionary mapping protocol or protocol and [e2e-llm-inference-service] hostname to the URL of the proxy. [e2e-llm-inference-service] :param hooks: (optional) Dictionary mapping hook name to one event or [e2e-llm-inference-service] list of events, event must be callable. [e2e-llm-inference-service] :param stream: (optional) whether to immediately download the response [e2e-llm-inference-service] content. Defaults to ``False``. [e2e-llm-inference-service] :param verify: (optional) Either a boolean, in which case it controls whether we verify [e2e-llm-inference-service] the server's TLS certificate, or a string, in which case it must be a path [e2e-llm-inference-service] to a CA bundle to use. Defaults to ``True``. When set to [e2e-llm-inference-service] ``False``, requests will accept any TLS certificate presented by [e2e-llm-inference-service] the server, and will ignore hostname mismatches and/or expired [e2e-llm-inference-service] certificates, which will make your application vulnerable to [e2e-llm-inference-service] man-in-the-middle (MitM) attacks. Setting verify to ``False`` [e2e-llm-inference-service] may be useful during local development or testing. [e2e-llm-inference-service] :param cert: (optional) if String, path to ssl client cert file (.pem). [e2e-llm-inference-service] If Tuple, ('cert', 'key') pair. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] # Create the Request. [e2e-llm-inference-service] req = Request( [e2e-llm-inference-service] method=method.upper(), [e2e-llm-inference-service] url=url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] files=files, [e2e-llm-inference-service] data=data or {}, [e2e-llm-inference-service] json=json, [e2e-llm-inference-service] params=params or {}, [e2e-llm-inference-service] auth=auth, [e2e-llm-inference-service] cookies=cookies, [e2e-llm-inference-service] hooks=hooks, [e2e-llm-inference-service] ) [e2e-llm-inference-service] prep = self.prepare_request(req) [e2e-llm-inference-service] [e2e-llm-inference-service] proxies = proxies or {} [e2e-llm-inference-service] [e2e-llm-inference-service] settings = self.merge_environment_settings( [e2e-llm-inference-service] prep.url, proxies, stream, verify, cert [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Send the request. [e2e-llm-inference-service] send_kwargs = { [e2e-llm-inference-service] "timeout": timeout, [e2e-llm-inference-service] "allow_redirects": allow_redirects, [e2e-llm-inference-service] } [e2e-llm-inference-service] send_kwargs.update(settings) [e2e-llm-inference-service] > resp = self.send(prep, **send_kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/sessions.py:589: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] request = [e2e-llm-inference-service] kwargs = {'cert': None, 'proxies': OrderedDict(), 'stream': False, 'timeout': 60, ...} [e2e-llm-inference-service] allow_redirects = True, stream = False, hooks = {'response': []} [e2e-llm-inference-service] adapter = [e2e-llm-inference-service] start = 1776970438.418253 [e2e-llm-inference-service] [e2e-llm-inference-service] def send(self, request, **kwargs): [e2e-llm-inference-service] """Send a given PreparedRequest. [e2e-llm-inference-service] [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] # Set defaults that the hooks can utilize to ensure they always have [e2e-llm-inference-service] # the correct parameters to reproduce the previous request. [e2e-llm-inference-service] kwargs.setdefault("stream", self.stream) [e2e-llm-inference-service] kwargs.setdefault("verify", self.verify) [e2e-llm-inference-service] kwargs.setdefault("cert", self.cert) [e2e-llm-inference-service] if "proxies" not in kwargs: [e2e-llm-inference-service] kwargs["proxies"] = resolve_proxies(request, self.proxies, self.trust_env) [e2e-llm-inference-service] [e2e-llm-inference-service] # It's possible that users might accidentally send a Request object. [e2e-llm-inference-service] # Guard against that specific failure case. [e2e-llm-inference-service] if isinstance(request, Request): [e2e-llm-inference-service] raise ValueError("You can only send PreparedRequests.") [e2e-llm-inference-service] [e2e-llm-inference-service] # Set up variables needed for resolve_redirects and dispatching of hooks [e2e-llm-inference-service] allow_redirects = kwargs.pop("allow_redirects", True) [e2e-llm-inference-service] stream = kwargs.get("stream") [e2e-llm-inference-service] hooks = request.hooks [e2e-llm-inference-service] [e2e-llm-inference-service] # Get the appropriate adapter to use [e2e-llm-inference-service] adapter = self.get_adapter(url=request.url) [e2e-llm-inference-service] [e2e-llm-inference-service] # Start time (approximately) of the request [e2e-llm-inference-service] start = preferred_clock() [e2e-llm-inference-service] [e2e-llm-inference-service] # Send the request [e2e-llm-inference-service] > r = adapter.send(request, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/sessions.py:703: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] request = , stream = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), verify = '/tmp/ca.crt' [e2e-llm-inference-service] cert = None, proxies = OrderedDict() [e2e-llm-inference-service] [e2e-llm-inference-service] def send( [e2e-llm-inference-service] self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Sends PreparedRequest object. Returns Response object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param request: The :class:`PreparedRequest ` being sent. [e2e-llm-inference-service] :param stream: (optional) Whether to stream the request content. [e2e-llm-inference-service] :param timeout: (optional) How long to wait for the server to send [e2e-llm-inference-service] data before giving up, as a float, or a :ref:`(connect timeout, [e2e-llm-inference-service] read timeout) ` tuple. [e2e-llm-inference-service] :type timeout: float or tuple or urllib3 Timeout object [e2e-llm-inference-service] :param verify: (optional) Either a boolean, in which case it controls whether [e2e-llm-inference-service] we verify the server's TLS certificate, or a string, in which case it [e2e-llm-inference-service] must be a path to a CA bundle to use [e2e-llm-inference-service] :param cert: (optional) Any user-provided SSL certificate to be trusted. [e2e-llm-inference-service] :param proxies: (optional) The proxies dictionary to apply to the request. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] conn = self.get_connection_with_tls_context( [e2e-llm-inference-service] request, verify, proxies=proxies, cert=cert [e2e-llm-inference-service] ) [e2e-llm-inference-service] except LocationValueError as e: [e2e-llm-inference-service] raise InvalidURL(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] self.cert_verify(conn, request.url, verify, cert) [e2e-llm-inference-service] url = self.request_url(request, proxies) [e2e-llm-inference-service] self.add_headers( [e2e-llm-inference-service] request, [e2e-llm-inference-service] stream=stream, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] verify=verify, [e2e-llm-inference-service] cert=cert, [e2e-llm-inference-service] proxies=proxies, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] chunked = not (request.body is None or "Content-Length" in request.headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(timeout, tuple): [e2e-llm-inference-service] try: [e2e-llm-inference-service] connect, read = timeout [e2e-llm-inference-service] timeout = TimeoutSauce(connect=connect, read=read) [e2e-llm-inference-service] except ValueError: [e2e-llm-inference-service] raise ValueError( [e2e-llm-inference-service] f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, " [e2e-llm-inference-service] f"or a single float to set both timeouts to the same value." [e2e-llm-inference-service] ) [e2e-llm-inference-service] elif isinstance(timeout, TimeoutSauce): [e2e-llm-inference-service] pass [e2e-llm-inference-service] else: [e2e-llm-inference-service] timeout = TimeoutSauce(connect=timeout, read=timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] resp = conn.urlopen( [e2e-llm-inference-service] method=request.method, [e2e-llm-inference-service] url=url, [e2e-llm-inference-service] body=request.body, [e2e-llm-inference-service] headers=request.headers, [e2e-llm-inference-service] redirect=False, [e2e-llm-inference-service] assert_same_host=False, [e2e-llm-inference-service] preload_content=False, [e2e-llm-inference-service] decode_content=False, [e2e-llm-inference-service] retries=self.max_retries, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] except (ProtocolError, OSError) as err: [e2e-llm-inference-service] raise ConnectionError(err, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] except MaxRetryError as e: [e2e-llm-inference-service] if isinstance(e.reason, ConnectTimeoutError): [e2e-llm-inference-service] # TODO: Remove this in 3.0.0: see #2811 [e2e-llm-inference-service] if not isinstance(e.reason, NewConnectionError): [e2e-llm-inference-service] raise ConnectTimeout(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(e.reason, ResponseError): [e2e-llm-inference-service] raise RetryError(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(e.reason, _ProxyError): [e2e-llm-inference-service] raise ProxyError(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(e.reason, _SSLError): [e2e-llm-inference-service] # This branch is for urllib3 v1.22 and later. [e2e-llm-inference-service] raise SSLError(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] > raise ConnectionError(e, request=request) [e2e-llm-inference-service] E requests.exceptions.ConnectionError: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/adapters.py:700: ConnectionError [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-with-configmap-ref', 'workload-llmd-simulator'], prompt='KServe is a'... {'name': 'workload-llmd-simulator-schedul-b2f159da'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] > wait_for_model_response(kserve_client, test_case, test_case.wait_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:413: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'scheduler-wi... {'name': 'workload-llmd-simulator-schedul-b2f159da'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-04-23T18:41:40.083642', start_time = 1776969700.0839787 [e2e-llm-inference-service] duration = 1102.439614534378, timestamp_end = '2026-04-23T19:00:02.523595' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-with-configmap-ref', 'workload-llmd-simulator'], prompt='KServe is a'... {'name': 'workload-llmd-simulator-schedul-b2f159da'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900, extra_headers = None [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_model_responds(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name, [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.response_assertion(response) [e2e-llm-inference-service] return response.text[: test_case.max_tokens] [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_model_responds, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:535: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_model_responds at 0x7fb8ef96f1a0> [e2e-llm-inference-service] timeout = 900, interval = 5.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:628: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_model_responds(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name, [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] > raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] E AssertionError: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:528: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1107 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1107 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1151 Created ConfigMap scheduler-config-e2e in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1056 Checking LLMInferenceServiceConfig router-managed-scheduler-config-305f7a8b in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1082 Resource not found, creating LLMInferenceServiceConfig router-managed-scheduler-config-305f7a8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1092 ✓ Successfully created LLMInferenceServiceConfig router-managed-scheduler-config-305f7a8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1056 Checking LLMInferenceServiceConfig scheduler-with-configmap-ref-sc-67492bcd in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1082 Resource not found, creating LLMInferenceServiceConfig scheduler-with-configmap-ref-sc-67492bcd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1092 ✓ Successfully created LLMInferenceServiceConfig scheduler-with-configmap-ref-sc-67492bcd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1056 Checking LLMInferenceServiceConfig workload-llmd-simulator-schedul-b2f159da in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1082 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-schedul-b2f159da [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1092 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-schedul-b2f159da [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-04-23T18:40:57.281656] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'scheduler-with-configmap-ref', 'workload-llmd-simulator'], prompt='KServe is a', service_name='scheduler-configmap-ref-test', endpoint='/v1/completions', max_tokens=100, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, before_test=[], after_test=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-configmap-ref-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-config-305f7a8b'}, [e2e-llm-inference-service] {'name': 'scheduler-with-configmap-ref-sc-67492bcd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-b2f159da'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1107 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-04-23T18:40:57.294398] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-configmap-ref-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-config-305f7a8b'}, [e2e-llm-inference-service] {'name': 'scheduler-with-configmap-ref-sc-67492bcd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-b2f159da'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-04-23T18:40:57.379643] end - ✅ in 0.085s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-04-23T18:40:57.379800] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-configmap-ref-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-config-305f7a8b'}, [e2e-llm-inference-service] {'name': 'scheduler-with-configmap-ref-sc-67492bcd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-b2f159da'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:57.380069] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:57.473689] end - ✅ in 0.094s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:58.473987] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:58.481632] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:40:59.481988] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:40:59.489638] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:00.489887] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:00.496849] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:01.497147] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:01.573488] end - ✅ in 0.076s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:02.573753] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:02.581523] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:03.581773] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:03.588991] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:04.589245] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:04.596299] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:05.596595] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:05.614086] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:06.614345] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:06.674205] end - ✅ in 0.060s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:07.674477] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:07.775828] end - ✅ in 0.101s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:08.776160] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:08.785772] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:09.786122] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:09.794890] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:10.795200] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:10.804350] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:11.804607] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:11.813000] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:12.813310] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:12.821420] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:13.821707] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:13.829466] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:14.829721] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:14.837354] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:15.837635] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:15.846700] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:16.846996] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:16.855365] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:17.855724] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:17.864204] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:18.864489] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:18.872380] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:19.872667] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:19.883360] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:20.883633] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:20.891831] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:21.892196] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:21.900471] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:22.900742] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:22.909742] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:23.909993] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:23.917990] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:24.918272] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:24.927232] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:25.927522] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:25.935374] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:26.935686] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:26.944878] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:27.945215] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:27.953786] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:28.954076] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:28.962332] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:29.962637] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:29.971338] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:30.971741] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:30.979801] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:31.980217] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:31.988687] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:32.988969] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:32.996872] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:33.997191] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:34.005537] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:35.005991] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:35.015127] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:36.015457] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:36.027608] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:37.028083] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:37.035674] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:38.035981] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:38.067615] end - ✅ in 0.031s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:39.068115] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:39.075460] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-04-23T18:41:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-04-23T18:41:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:40.075757] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:40.083465] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-04-23T18:41:40.083575] end - ✅ in 42.704s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-04-23T18:41:40.083642] start - args=(, TestCase(base_refs=['router-managed', 'scheduler-with-configmap-ref', 'workload-llmd-simulator'], prompt='KServe is a', service_name='scheduler-configmap-ref-test', endpoint='/v1/completions', max_tokens=100, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, before_test=[], after_test=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-configmap-ref-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-config-305f7a8b'}, [e2e-llm-inference-service] {'name': 'scheduler-with-configmap-ref-sc-67492bcd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-b2f159da'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-04-23T18:41:40.083983] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-configmap-ref-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-config-305f7a8b'}, [e2e-llm-inference-service] {'name': 'scheduler-with-configmap-ref-sc-67492bcd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-b2f159da'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:41:40.084301] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:41:40.091270] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-04-23T18:41:40.091338] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:518 Calling LLM service at http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 100} [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] ERROR e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:527 ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-04-23T18:47:49.215138] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-configmap-ref-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-config-305f7a8b'}, [e2e-llm-inference-service] {'name': 'scheduler-with-configmap-ref-sc-67492bcd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-b2f159da'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:47:49.215638] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:47:49.250034] end - ✅ in 0.034s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-04-23T18:47:49.250203] end - ✅ in 0.035s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:518 Calling LLM service at http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 100} [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] ERROR e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:527 ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:632 Waiting: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-04-23T18:53:58.375676] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-configmap-ref-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-config-305f7a8b'}, [e2e-llm-inference-service] {'name': 'scheduler-with-configmap-ref-sc-67492bcd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-b2f159da'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T18:53:58.376267] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test', 'v1alpha1'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T18:53:58.417251] end - ✅ in 0.041s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-04-23T18:53:58.417424] end - ✅ in 0.041s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:518 Calling LLM service at http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 100} [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")': /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions [e2e-llm-inference-service] ERROR e2e.llmisvc.test_llm_inference_service:test_llm_inference_service.py:527 ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-04-23T19:00:02.523595] end - ❌ 1102.440s: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llmisvc] [2026-04-23T19:00:02.523669] start - args=(, 'scheduler-configmap-ref-test', 'kserve-ci-e2e-test'), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llmisvc] [2026-04-23T19:00:02.559120] end - ✅ in 0.035s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-04-23T19:00:03.863408] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-configmap-ref-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-config-305f7a8b'}, [e2e-llm-inference-service] {'name': 'scheduler-with-configmap-ref-sc-67492bcd'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-schedul-b2f159da'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-04-23T19:00:03.884319] end - ✅ in 0.021s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-04-23T19:00:03.884450] end - ❌ 1146.602s: ❌ Failed to call model: HTTPConnectionPool(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-configmap-ref-test/v1/completions (Caused by NameResolutionError("HTTPConnection(host='openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local', port=80): Failed to resolve 'openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local' ([Errno -2] Name or service not known)")) [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1107 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1178 Deleted ConfigMap scheduler-config-e2e from namespace kserve-ci-e2e-test [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:151 [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:151 [e2e-llm-inference-service] /workspace/source/test/e2e/llmisvc/test_llm_inference_service.py:151: PytestUnknownMarkWarning: Unknown pytest.mark.custom_gateway - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:200 [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:200 [e2e-llm-inference-service] /workspace/source/test/e2e/llmisvc/test_llm_inference_service.py:200: PytestUnknownMarkWarning: Unknown pytest.mark.custom_gateway - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:252 [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:252 [e2e-llm-inference-service] /workspace/source/test/e2e/llmisvc/test_llm_inference_service.py:252: PytestUnknownMarkWarning: Unknown pytest.mark.custom_gateway - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:299 [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:299 [e2e-llm-inference-service] /workspace/source/test/e2e/llmisvc/test_llm_inference_service.py:299: PytestUnknownMarkWarning: Unknown pytest.mark.cluster_gpu - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:316 [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:316 [e2e-llm-inference-service] /workspace/source/test/e2e/llmisvc/test_llm_inference_service.py:316: PytestUnknownMarkWarning: Unknown pytest.mark.no_scheduler - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:329 [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:329 [e2e-llm-inference-service] /workspace/source/test/e2e/llmisvc/test_llm_inference_service.py:329: PytestUnknownMarkWarning: Unknown pytest.mark.cluster_multi_node - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:124: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:124: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:124: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:124: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:124: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:124: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:124: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:124: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:124: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:124: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:124: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:124: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:124: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] ====== 4 failed, 18 passed, 3 skipped, 28 warnings in 3290.78s (0:54:50) ======= [must-gather] [must-gather ] OUT 2026-04-23T19:16:20.939261479Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 4070c745-bd54-4f25-90de-26000850e63f [must-gather] ClientVersion: 4.21.5 [must-gather] ClusterVersion: Stable at "4.20.19" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-04-23T19:16:20.997836678Z namespace/openshift-must-gather-8n7dv created [must-gather] [must-gather ] OUT 2026-04-23T19:16:21.004227085Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-wbdn2 created [must-gather] [must-gather ] OUT 2026-04-23T19:16:21.032192092Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:25.650604028Z [disk usage checker] Started [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:25.684085877Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:25.934026474Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.107105447Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.107134607Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.107134607Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.110603943Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.281850058Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.281881712Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.281881712Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.284946194Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.451379830Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.451411279Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.451411279Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.454540945Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.623818934Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.623850931Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.623850931Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.626902326Z Error getting logs from rhods-notebooks [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.790539401Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.790569239Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.790569239Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.793553046Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.959417302Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.959448722Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.959448722Z namespaces "istio-system" not found [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:26.962525084Z Error getting logs from istio-system [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:27.127639725Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:27.127670245Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:27.127670245Z namespaces "knative-serving" not found [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:27.130700391Z Error getting logs from knative-serving [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:27.303868965Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:27.303970968Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:27.303970968Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:27.307194046Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:27.602197976Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:27.908862453Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:28.057465592Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:28.147374773Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:28.150406622Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:28.288205115Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:28.375357363Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:28.378501911Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:28.514478244Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:28.597507926Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:28.600409903Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:28.738458233Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:28.822488135Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:28.825429696Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:28.963862197Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:29.050944014Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:29.054054555Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:29.191275119Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:29.275237682Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:29.278063234Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:29.410405615Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:29.494017621Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:29.496936190Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:29.632489828Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:29.714959111Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:29.717711539Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:29.851080833Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:29.936827031Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:29.939768547Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:30.074834550Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:30.157627384Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:30.160462016Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:30.295063302Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:30.380832281Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:30.383889929Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:30.521175919Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:30.603937567Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:30.606778881Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:30.691126076Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:30.744374144Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:30.825216363Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:30.828119266Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:30.963894960Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:31.049550515Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:31.052581743Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:31.188609861Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:31.271375748Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:31.274328624Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:31.411944790Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:31.493900180Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:31.496802325Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:31.638651522Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:31.723004876Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:31.726052021Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:31.866852389Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:31.951332008Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:31.954223205Z Error collecting info from [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:32.696427849Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:32.830750515Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:32.973713146Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:33.107827030Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:33.243898015Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:33.853747560Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:33.988097882Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:34.132995044Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:34.271694763Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:34.403440430Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:34.949972608Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:35.280788577Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:35.681111602Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:35.696891853Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:35.891179427Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:36.058396542Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:36.341635064Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:36.752286361Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:36.971257678Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:37.144731019Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:37.438762030Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:37.824922686Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:38.048014195Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:38.223435004Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:38.642256075Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:39.031917553Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:39.246211796Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:39.419293275Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:39.728578232Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:40.125854332Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:40.334660189Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:40.502809403Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:40.703619965Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:40.795818149Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:41.184218916Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:41.387406631Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:41.561498529Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:41.893726157Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:42.289769276Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:42.502125091Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:42.680056980Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:42.974784871Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:43.406899834Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:43.631887720Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:43.802353257Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:44.096237709Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:44.491424094Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:44.701948007Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:44.867894345Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:45.157654057Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:45.547751061Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:45.710820675Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:45.761658364Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:45.931470549Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:46.279994227Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:46.687801931Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:46.891452619Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:47.061786968Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:47.365567349Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:47.754651368Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:47.973857904Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:48.143582429Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:48.427240258Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:48.809691225Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:49.024711521Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:49.193218410Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:49.478165308Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:49.857673295Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:50.070859046Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:50.239201194Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:50.532168416Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:50.715620017Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:50.717466944Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:50.961730308Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:51.132621429Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:51.413019059Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:51.592155039Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:51.835090210Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:52.004730322Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:52.407479816Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:52.612691036Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:52.872233352Z Wrote inspect data to must-gather. [must-gather] [must-gather-cqx24] POD 2026-04-23T19:16:52.897482382Z Caches written to disk [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:53.909864758Z waiting for gather to complete [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:53.913969208Z downloading gather output [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.183100905Z receiving incremental file list [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.200196936Z ./ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.20038954Z aggregated-discovery-api.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.200746688Z aggregated-discovery-apis.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.201492684Z event-filter.html [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.202950206Z timestamp [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.203084819Z version [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.206892342Z cluster-scoped-resources/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.206920512Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.206927362Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.206977313Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.207070385Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.207079206Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.207122006Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.207209268Z namespaces/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.207218318Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.20727255Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.207344511Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.207395962Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.207453314Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.207491815Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.207590377Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.207788051Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.208003355Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.208053576Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.208105198Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.208148439Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.20819425Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.208286802Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.208337553Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.208393494Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.208499836Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.208541527Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.208585938Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.208719361Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.208820783Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.211480421Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.211566573Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.211960371Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.212096254Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.212294168Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.21237039Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.212417831Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.212492823Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.212529604Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.212590335Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.212632786Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.212721838Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.212766669Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.212800979Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.212858181Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.212893501Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.212999534Z namespaces/kserve-ci-e2e-test/pods/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.213013934Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-6c59fbf55c-mqqmd/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.213047445Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-6c59fbf55c-mqqmd/router-gateway-1-openshift-default-6c59fbf55c-mqqmd.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.213171508Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-6c59fbf55c-mqqmd/istio-proxy/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.213179668Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-6c59fbf55c-mqqmd/istio-proxy/istio-proxy/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.213183288Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-6c59fbf55c-mqqmd/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.213230639Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-6c59fbf55c-mqqmd/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.213407193Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-6c59fbf55c-mqqmd/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.213481294Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-6c59fbf55c-mqqmd/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.213530246Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-6866b85949-kqbbn/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.213595887Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-6866b85949-kqbbn/router-gateway-2-openshift-default-6866b85949-kqbbn.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.2137352Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-6866b85949-kqbbn/istio-proxy/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.21374578Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-6866b85949-kqbbn/istio-proxy/istio-proxy/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.21375176Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-6866b85949-kqbbn/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.213780961Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-6866b85949-kqbbn/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.213933584Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-6866b85949-kqbbn/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.214029266Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-6866b85949-kqbbn/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.214070167Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.214116578Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.214314083Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/llm-d-routing-sidecar/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.214321683Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/llm-d-routing-sidecar/llm-d-routing-sidecar/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.214325293Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.214365453Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.214506617Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.214584648Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.214625189Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/main/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.214632969Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/main/main/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.214636359Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/main/main/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.21467262Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/main/main/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.215073309Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/main/main/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.21515071Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/main/main/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.215191002Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/storage-initializer/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.215201062Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/storage-initializer/storage-initializer/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.215210262Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.215232112Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.215407346Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.215489268Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-d9f6bb5b8-pckhm/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.215534589Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.215567459Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.215691332Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk/main/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.215700502Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk/main/main/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.215706283Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk/main/main/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.215739393Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk/main/main/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.215963118Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk/main/main/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.21603827Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk/main/main/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.216076011Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk/storage-initializer/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.216083941Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk/storage-initializer/storage-initializer/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.216089251Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.216126322Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.216287195Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.216397588Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-7d9f7645f-glgwk/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.216445319Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.21651301Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.216575552Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.216622453Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.216669074Z namespaces/kuadrant-system/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.216785136Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.216859467Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.21695295Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.217006921Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.217063352Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.217146734Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.217493241Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.217817208Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.217864149Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.21791155Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.217990112Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.218038403Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.218129265Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.218176176Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.218225777Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.218306019Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.2183561Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.218400531Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.218545974Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.218664287Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.219135537Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.219223049Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.219595807Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.21972757Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.219968785Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220106098Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220166699Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220296182Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220354173Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220405475Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220454085Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220550038Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220588319Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.22063272Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.22067441Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220730742Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220776112Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220784143Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220834674Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/authorino-68bd676465-vlxjh.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220913346Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/authorino/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220928276Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/authorino/authorino/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220941616Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/authorino/authorino/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.220993537Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/authorino/authorino/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.221872516Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.221976219Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/authorino/authorino/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.222009119Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.22205994Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/authorino-operator-7587b89b76-hnn8l.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.222155722Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/manager/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.222165873Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/manager/manager/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.222170873Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/manager/manager/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.222202143Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/manager/manager/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.222386447Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.222461869Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/manager/manager/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.2225063Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.222545701Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/dns-operator-controller-manager-844548ff4c-hhjsv.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.222657753Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/manager/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.222677114Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/manager/manager/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.222682744Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/manager/manager/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.222709694Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/manager/manager/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.222825017Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.222896389Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/manager/manager/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.22295935Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.223005741Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/kuadrant-console-plugin-6c886788f8-mj4k9.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.223082493Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/kuadrant-console-plugin/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.223091013Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.223095203Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.223135974Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.223216325Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.223287567Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.223332198Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.223373799Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/kuadrant-operator-controller-manager-6ddf9554fc-s54pc.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.223479001Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/manager/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.223485841Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/manager/manager/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.223489411Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/manager/manager/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.223523992Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/manager/manager/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.224670387Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.224739749Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/manager/manager/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.22478653Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.22482028Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/limitador-limitador-67566c68b4-52nnx.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.224918532Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/limitador/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.224930393Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/limitador/limitador/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.224933853Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/limitador/limitador/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.224994094Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/limitador/limitador/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.225084156Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.225185928Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/limitador/limitador/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.225206859Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.22526316Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/limitador-operator-controller-manager-c7fb4c8d5-l5476.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.225347722Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/manager/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.225356412Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/manager/manager/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.225363122Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/manager/manager/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.225396153Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/manager/manager/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.225570447Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.225643368Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/manager/manager/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.225690679Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.22571514Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.225772511Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.225819992Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.225877013Z namespaces/openshift-ingress/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.225945055Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.226006376Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.226055187Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.226103978Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.226153149Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.226237991Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.226475856Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.226725602Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.226771423Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.226819704Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.226883955Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.226963337Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.227079289Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.22712685Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.227174942Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.227252713Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.227304044Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.227343445Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.227762174Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.227863556Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.228152812Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.228234464Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.228475419Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.228606273Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.228840498Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.22895174Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229019131Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229101573Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229127284Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229189045Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229235326Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229324018Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229367909Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.22941585Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229469481Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229518592Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229559183Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229566033Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229612314Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/istiod-openshift-gateway-55ff986f96-fjvbp.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229733157Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/discovery/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229740087Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/discovery/discovery/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229743647Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/discovery/discovery/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.229780718Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/discovery/discovery/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.231581307Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.231661559Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/discovery/discovery/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.231693729Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.231746701Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.231859383Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/istio-proxy/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.231867553Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/istio-proxy/istio-proxy/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.231871783Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.231931494Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.232110158Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.23217925Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.232213041Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.232260792Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/router-default-747f569c94-mjbpd.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.232365814Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/router/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.232373144Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/router/router/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.232376544Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/router/router/logs/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.232400555Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/router/router/logs/current.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.232515177Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/router/router/logs/previous.insecure.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.232594999Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/router/router/logs/previous.log [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.23262434Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.232676511Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.232728952Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.232780863Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.239892057Z [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.239926528Z sent 3,446 bytes received 393,328 bytes 264,516.00 bytes/sec [must-gather] [must-gather-cqx24] OUT 2026-04-23T19:16:54.239936878Z total size is 4,804,304 speedup is 12.11 [must-gather] [must-gather ] OUT 2026-04-23T19:16:54.385097957Z namespace/openshift-must-gather-8n7dv deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 4070c745-bd54-4f25-90de-26000850e63f [must-gather] ClientVersion: 4.21.5 [must-gather] ClusterVersion: Stable at "4.20.19" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-04-23T19:16:54.512527021Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd19538052f0deb24938546fa0d17135330102ff01b5699610d2fdf82785eaab [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 4070c745-bd54-4f25-90de-26000850e63f [must-gather] ClientVersion: 4.21.5 [must-gather] ClusterVersion: Stable at "4.20.19" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-04-23T19:16:54.543137215Z namespace/openshift-must-gather-j4sk6 created [must-gather] [must-gather ] OUT 2026-04-23T19:16:54.552755494Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-kzst6 created [must-gather] [must-gather ] OUT 2026-04-23T19:16:54.578464072Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd19538052f0deb24938546fa0d17135330102ff01b5699610d2fdf82785eaab created [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:56.085247764Z [disk usage checker] Started [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:56.088645362Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:56.329396687Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:56.478044197Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:56.597257452Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:56.706476031Z Gathering data for ns/default... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:57.129252296Z Gathering data for ns/openshift... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:57.633183028Z Gathering data for ns/kube-system... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.438739667Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.706297044Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.717078732Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.717468786Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.718624179Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.727788793Z INFO: Gathering HAProxy config files [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.734449345Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.741836003Z INFO: Collecting host service logs for crio [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.742263257Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.742606901Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.742944902Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.743301847Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.743641746Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.743976582Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.744325214Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.744655424Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.745028232Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.745374471Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:58.745775849Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.067937788Z No resources found [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.147418731Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.147508223Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.147508223Z namespaces "assisted-installer" not found [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.216438542Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.306808230Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-6gf74 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.306848335Z insights-runtime-extractor-cl2hj [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.306867812Z insights-runtime-extractor-dpxlz [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.306923697Z insights-runtime-extractor-wsvkb [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.606614010Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.606707268Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.694151447Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.815955658Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.815955658Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:16:59.883772993Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:00.182956412Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:00.660962122Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:00.779325990Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:00.829253358Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:00.976372550Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.014283339Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.028489640Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.118194210Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.155840580Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.198240057Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.373725986Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.442350058Z No resources found [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.491064047Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.493936759Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.495068661Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.507926659Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.534596922Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.580429034Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.586388142Z tar: Removing leading `/' from member names [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.594864879Z INFO: HAProxy config collection complete. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.678295071Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.691178328Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.807781472Z INFO: 'previous-logs' folder not found on ip-10-0-137-68.ec2.internal, skipping... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:01.908364451Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.117579356Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.174269976Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.216816013Z INFO: 'previous-logs' folder not found on ip-10-0-138-68.ec2.internal, skipping... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.228687616Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.278621317Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.471342433Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.535505846Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.611614331Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.644179451Z INFO: 'previous-logs' folder not found on ip-10-0-143-218.ec2.internal, skipping... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.644226318Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.644250139Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.744599374Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.894732938Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.936264498Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:02.952586454Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.244243429Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.296854736Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.326787720Z INFO: INTERCONNECT MODE [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.326879775Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.545742755Z INFO: Gathering OVN_Northbound from ovnkube-node-b27s8... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.546012590Z INFO: Gathering OVN_Northbound from ovnkube-node-p7h4b... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.546279664Z INFO: Gathering OVN_Northbound from ovnkube-node-swcqx... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.546714008Z INFO: Gathering OVN_Southbound from ovnkube-node-b27s8... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.547038995Z INFO: Gathering OVN_Southbound from ovnkube-node-p7h4b... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.547353293Z INFO: Gathering OVN_Southbound from ovnkube-node-swcqx... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.569227919Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.590504383Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.848074756Z tar: Removing leading `/' from member names [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.857185893Z tar: Removing leading `/' from member names [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.899711684Z tar: Removing leading `/' from member names [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:03.956604523Z tar: Removing leading `/' from member names [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.011117349Z tar: Removing leading `/' from member names [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.014668649Z tar: Removing leading `/' from member names [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.092132480Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.107583109Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.397868717Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.467669545Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.682365379Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.682412821Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.683015076Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.685358123Z 5.2M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.688387440Z ovnk_database_store/ [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.688387440Z ovnk_database_store/ovnkube-node-b27s8_nbdb [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.703228499Z ovnk_database_store/ovnkube-node-p7h4b_nbdb [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.715064238Z ovnk_database_store/ovnkube-node-swcqx_nbdb [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.729003185Z ovnk_database_store/ovnkube-node-swcqx_sbdb [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.760631296Z ovnk_database_store/ovnkube-node-b27s8_sbdb [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.769707147Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.791999972Z ovnk_database_store/ovnkube-node-p7h4b_sbdb [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.829678025Z INFO: Network log collection complete. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:04.903828182Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:05.075241862Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:05.391537969Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:05.394655996Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:05.508262933Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:05.979293224Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:06.048745641Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:06.142854914Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:06.169514022Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:06.293453343Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:06.303570822Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:06.518302788Z Gathering data for ns/openshift-console... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:07.032059682Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:07.438332720Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:07.460055513Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:07.825314018Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:08.573943062Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:08.580834330Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:09.119464277Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:09.539285180Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:09.703489559Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:10.207568202Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:10.698227830Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:10.830809564Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:11.177243736Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:11.959275398Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:13.104162365Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:13.579663955Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:14.176055791Z Gathering data for ns/kserve... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:14.240202718Z Waiting for performance profile collector pods to become ready: 8 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:15.375665072Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:15.647215268Z Collecting performance related data for node ip-10-0-138-68.ec2.internal [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:15.654412835Z Collecting performance related data for node ip-10-0-137-68.ec2.internal [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:15.662369036Z Collecting performance related data for node ip-10-0-143-218.ec2.internal [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:16.074888421Z Gathering data for ns/openshift-config... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:16.182450758Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:16.470667791Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:17.113192118Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:17.528013402Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:17.908572015Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:18.323162882Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:18.731650727Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:19.131217786Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:19.513541800Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:19.925525117Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:20.324663224Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:21.027724609Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:21.187161657Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:22.698393011Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:23.883012344Z Collecting kubelet logs for node ip-10-0-138-68.ec2.internal [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:23.884862056Z Collecting kubelet logs for node ip-10-0-137-68.ec2.internal [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:23.886577516Z Collecting kubelet logs for node ip-10-0-143-218.ec2.internal [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:24.315582412Z daemonset.apps "perf-node-gather-daemonset" deleted [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:24.321651418Z INFO: Node performance data collection complete. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:24.515112307Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:24.908046256Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:25.518222295Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:26.027648229Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:26.192731823Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:26.481812996Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:26.828461389Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:27.360203964Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:27.733864519Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:28.213399536Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:28.532518968Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:28.866433537Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:29.587804532Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:29.976597926Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:30.345854609Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:30.771452054Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:31.199215114Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:31.767637688Z Wrote inspect data to must-gather. [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:31.767680984Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:31.767680984Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:31.913585186Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-gr2p9] POD 2026-04-23T19:17:32.262039846Z Caches written to disk [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.270960317Z waiting for gather to complete [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.274465243Z downloading gather output [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.538996269Z receiving incremental file list [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.560705812Z ./ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.560822515Z aggregated-discovery-api.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.561189284Z aggregated-discovery-apis.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.561991814Z event-filter.html [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.564639649Z timestamp [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.564761792Z version [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.573956758Z cluster-scoped-resources/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.573973258Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.573978209Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.57403248Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.574150363Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.574264396Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.574317597Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.574375479Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.574459821Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.574552913Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.574668186Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.574708397Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.574829599Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.574918142Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.575287121Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.575336522Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.575556788Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.575838244Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.57606437Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.576215694Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.579523245Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.581090084Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.581227997Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.58136533Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.581834472Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.582003706Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.582155Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.582268013Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.583036472Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.583270087Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.5833966Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.583564914Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.584835786Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.585116283Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.585289577Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.585454911Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.585734488Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.585874251Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.586103447Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.586366903Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.587105821Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.587228884Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.587348818Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.58745908Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.587562943Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.587670565Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.58783469Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.588043964Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.588812503Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.588975998Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.589165392Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.591006928Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.591175202Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.591331156Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.591549161Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.591807627Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.592248768Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.592370321Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.592495814Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.592607607Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.59272198Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.592845013Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.592992166Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.593176901Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.593320204Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.593431057Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.593691274Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.593827897Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.5939708Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.594516144Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.594644877Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.59477328Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.595034276Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.595765525Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.595897668Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.596116493Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.596243476Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.596419221Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.596570864Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.596735168Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.596870452Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.597029576Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.597154599Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.597281392Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.597415575Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.597532748Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.597676332Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.597823745Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.597972129Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.598131903Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.598494722Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.598669466Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.599068856Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.599189769Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.599996349Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.600116381Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.600225574Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.600350947Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.600527832Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.600786628Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.601253389Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.601361892Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.601521906Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.601647329Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.601766702Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.601921996Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.60207589Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.602195593Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.604376706Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.604830717Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.605450063Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.605630677Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.605779801Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.605924014Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.606084878Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.606196601Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.607039682Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocnis.sailoperator.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.60736616Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.607659067Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiorevisions.sailoperator.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.613453869Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiorevisiontags.sailoperator.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.613571182Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istios.sailoperator.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.614989437Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.61631167Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.616446713Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.616596857Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.61672747Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.616855573Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.617019537Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.61714612Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.625754162Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.62606119Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.630360655Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.634462876Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.63462427Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigpools.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.634824365Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.635120632Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.635285666Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.63543564Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.635546833Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.635702487Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.635978713Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.636095516Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.636204929Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.636409604Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.636539307Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.63666184Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.636779963Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.636895486Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.63707402Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.637293616Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.637436239Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.637560992Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.637683905Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.637819549Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.637951062Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.638092066Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.638202048Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.638324441Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.638576357Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.63870267Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.638974567Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.63910605Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.639209733Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.639328726Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.641261763Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.641384526Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.641495599Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.641605362Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.641709394Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.641858918Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.642020172Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.642171116Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.642289549Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.643791626Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.6439723Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.644093053Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.644246747Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.64436649Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.644546184Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.64479644Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.645328613Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.64560709Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.645729483Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.645841376Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.645976879Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.646570524Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.646793689Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.646916672Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.648323277Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.648494151Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.648639685Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.648749108Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.648928222Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.649215689Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.649343582Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.649504936Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.649928546Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.65007666Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.650191843Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.650362447Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.650521631Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.650653064Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.650793198Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.651029784Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ztunnels.sailoperator.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.652059559Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.652284215Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.652343266Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.652441988Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.652533961Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.652639973Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.652728775Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.65292631Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.653071194Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.653187397Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.653281169Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.653375231Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.653469204Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.653559736Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.653652188Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.65375018Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.653833613Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.653933125Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.654034107Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.65412713Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.654210832Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.654308894Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.654402227Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.654483618Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.654576361Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.654663193Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.654762245Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.654847207Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.65496112Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.655058433Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.655148995Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.655239377Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.655325299Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.655417552Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.655504714Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.655593546Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.655709929Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.655816291Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.655932344Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.656049327Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.656146939Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.656258592Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.656368275Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.656458857Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.65655178Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.656668782Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.656763505Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.656857957Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.65699123Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.657148514Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.657273847Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.65739451Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.657517333Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.657623026Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.sailoperator.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.657719058Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.65780699Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.657957704Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.658081007Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.65820067Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.658299222Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.658398015Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.658492357Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.658584399Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.658685822Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.658786704Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.658888187Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.659061381Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.659167804Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.659252316Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.659349878Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.65944646Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.659542953Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.659631835Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.659733598Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.659866741Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.660013125Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.660113997Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.660208179Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.660303032Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.660412364Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.660513387Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.660614749Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.660707862Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.sailoperator.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.660803134Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.660909987Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.66104062Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.661138922Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.661237635Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.661330977Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.661426089Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.661545922Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.661666995Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.661782898Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.661922372Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.662090176Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.662215429Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.662335472Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.662455434Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.662570568Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.66267206Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.662768212Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.662863655Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.662993508Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.6630929Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta2.authorino.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.663185303Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.663315366Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.663422178Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.663542711Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.663593533Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.663789947Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.663930781Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.664040764Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.664139276Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.664456294Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.664575267Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.664666819Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.664750601Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.664863964Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.664988777Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.665082849Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.665164011Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.665258883Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.665339736Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.665441718Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.66552904Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.665627593Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.665721535Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.665808287Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.665893809Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.666006422Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.666090374Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.666185056Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.666233747Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.666364651Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.666476244Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.666569896Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.666664688Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.666759731Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.666876863Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.667083718Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.667212842Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.667318334Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.667431077Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.667523289Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.667656882Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.667751955Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.667853617Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.66797019Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.668064892Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.668158855Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.668251037Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.668342439Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.668440122Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.668510134Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.668556235Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.668639987Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.668681938Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.668738919Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.6687811Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.668859082Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.668923904Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.668995095Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.669040867Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.669096858Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.669135569Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.66919912Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.669215671Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.669265172Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.669373425Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.669449117Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.669501658Z cluster-scoped-resources/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.669510028Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.669541279Z cluster-scoped-resources/core/nodes/ip-10-0-137-68.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.669729104Z cluster-scoped-resources/core/nodes/ip-10-0-138-68.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.669872647Z cluster-scoped-resources/core/nodes/ip-10-0-143-218.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.67000865Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.670064022Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.670122963Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.670222436Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/endpoint-controller.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.670322968Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.670440261Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.670533803Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.670615385Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.670773069Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.670891232Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.671028325Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.671120798Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.671231701Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.671325303Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.671427315Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.671531848Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.671639251Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.671755443Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.671895897Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.672059021Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.672160653Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.672259486Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.672354648Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.67245628Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.672538213Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/workload-leader-election.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.672630525Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.672677416Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.672769058Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.67286099Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.673012214Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.673098056Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.673190038Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.673356743Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.673483096Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.673587298Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.673639029Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.67364951Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.673692271Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.673761323Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.673768233Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.673814934Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.673925047Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.673970418Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.674036279Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.67405044Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.674093741Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.674191993Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.674347727Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.674405999Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.674418959Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.67445633Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.674574643Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.674582953Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.674619384Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.674708056Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.674759757Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.6748747Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.674940232Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.674999923Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675050184Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675121296Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675171667Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675314351Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675372072Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675438914Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675491755Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675547416Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675597748Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675660349Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675711921Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675793903Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675814953Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675913656Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.675968147Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.676055919Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.67610586Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.676171192Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.676206693Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.676286095Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.676343116Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.676452739Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.676461629Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.67649847Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.676612973Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.676670514Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.676820558Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.676996982Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.677197907Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.67731175Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.677436233Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.677541576Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.677646328Z cluster-scoped-resources/operators.coreos.com/operators/servicemeshoperator3.openshift-operators.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.67773258Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.677750011Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.677800012Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.677890304Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.678063398Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.678221102Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.678325355Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.678408137Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.678502619Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.678601372Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.678682884Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.678783256Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.678873849Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.679001652Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.679109594Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.679197516Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.679287908Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.679379421Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.679473053Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.679564995Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.679667848Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.679714989Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.679818441Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.679914424Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.680040337Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.680205771Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.680304484Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.680422016Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.680515719Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.680603431Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.680699393Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.680792565Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.680887068Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681044692Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681140744Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681238036Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681325609Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681419791Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681522683Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681579035Z cluster-scoped-resources/sailoperator.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681588495Z cluster-scoped-resources/sailoperator.io/istios/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681636046Z cluster-scoped-resources/sailoperator.io/istios/openshift-gateway.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681703158Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681717018Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.6817712Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681832121Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681839641Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681869842Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681971154Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.681982295Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.682025486Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.682100108Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.682147359Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-137-68.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.682250551Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-138-68.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.682332403Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-143-218.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.682385305Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.682431066Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.682515378Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.682565389Z host_service_logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.6826058Z host_service_logs/masters/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.682643481Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.682721673Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.682788725Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.682849986Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.68301709Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.683131503Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.683218835Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.683296957Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.683381919Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.683458191Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.683545943Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.683582854Z ingress_controllers/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.683600115Z ingress_controllers/default/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.683605825Z ingress_controllers/default/router-default-747f569c94-mjbpd/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.683660256Z ingress_controllers/default/router-default-747f569c94-mjbpd/haproxy.config [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.683781749Z insights-data/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.683790019Z machine_config_ondisk/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.68379357Z machine_config_termination_logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.68380192Z monitoring/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.68381323Z monitoring/alertmanager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.683845991Z monitoring/alertmanager/status.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.683974444Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.684027725Z monitoring/prometheus/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.684083037Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.684177289Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.68423841Z monitoring/prometheus/rules.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.684993939Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.68504928Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.685123862Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.685971223Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.686019454Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.686070035Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.686169958Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.6862472Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.686343042Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.686403554Z monitoring/prometheus/status/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.686453445Z monitoring/prometheus/status/config.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.686888215Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.686996408Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.6870936Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.687254795Z namespaces/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.687261605Z namespaces/cert-manager-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.687267145Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.687271005Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.687315526Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.687374057Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.687386788Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.687438709Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.687666145Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.68788334Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.688098575Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.68830813Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.688569007Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.688845114Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.688888685Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-6wvsz.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.689085949Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.689124581Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.689196732Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.689243173Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/cert-manager-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.689308125Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.689349636Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.689426668Z namespaces/cert-manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.689436068Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.689441988Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.68949389Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.689660724Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.689841968Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.690023363Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.690228838Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.690504804Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.690768151Z namespaces/default/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.690826252Z namespaces/default/default.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.690916825Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.690978386Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.691045768Z namespaces/default/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.691100859Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.691185691Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.691267893Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.691351775Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.691402317Z namespaces/default/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.691448728Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.691496019Z namespaces/default/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.69154255Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.691622622Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.691690724Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.691740935Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.691824387Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.691911689Z namespaces/default/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.69197182Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.692089663Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.692180425Z namespaces/default/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.692387731Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.692478573Z namespaces/default/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.692561385Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.692693988Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.692861642Z namespaces/default/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.693000156Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.693044047Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.693128899Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.69317099Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.693221031Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.693266262Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.693384755Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.693476048Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.69356025Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.693607881Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.693669012Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.693712633Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.693763164Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.693769755Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.693807986Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.694007601Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.694189395Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.694330698Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.694622626Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.694865182Z namespaces/default/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.695095957Z namespaces/default/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.695145629Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.69519565Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.695233421Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.695293702Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.695300122Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.695303802Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.695354694Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/epp-kserve-ci-e2e-test-scheduler-ha-replicas-test-inference-pool.gateway-api-inference-extension.sigs.k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.695462196Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.695469126Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.695516088Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.69562517Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.695699912Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.695710683Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.695765354Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.695948748Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.696135933Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.696277056Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.696480711Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.696724957Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.697004554Z namespaces/kserve/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.697025655Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.697113957Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.697154578Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.69723219Z namespaces/kserve/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.697278351Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.697369663Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.697571438Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.697768963Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.697816534Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.697849975Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.697925277Z namespaces/kserve/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.69803532Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.698116622Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.698195193Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.698242815Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.698326467Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.698364938Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.698375238Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.698412019Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.698509181Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.698595383Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.698719826Z namespaces/kserve/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.698761227Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.699078625Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.699207938Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.699523986Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.699607798Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.699861534Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.700008478Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.700251904Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.700346837Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.700375557Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.70047592Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.70051137Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.700580502Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.700620213Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.700712915Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.700794317Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.700854669Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.70092132Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.701016613Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.701050564Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.701165646Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.701225218Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.701276889Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.701333711Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.701341441Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.701387502Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.701574327Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.701753761Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.701893844Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.70211323Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.702344846Z namespaces/kserve/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.702559471Z namespaces/kserve/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.702568201Z namespaces/kserve/pods/kserve-controller-manager-6fc5d867c5-mgfdv/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.702609922Z namespaces/kserve/pods/kserve-controller-manager-6fc5d867c5-mgfdv/kserve-controller-manager-6fc5d867c5-mgfdv.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.702718935Z namespaces/kserve/pods/kserve-controller-manager-6fc5d867c5-mgfdv/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.702726815Z namespaces/kserve/pods/kserve-controller-manager-6fc5d867c5-mgfdv/manager/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.702736065Z namespaces/kserve/pods/kserve-controller-manager-6fc5d867c5-mgfdv/manager/manager/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.702784166Z namespaces/kserve/pods/kserve-controller-manager-6fc5d867c5-mgfdv/manager/manager/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.702948441Z namespaces/kserve/pods/kserve-controller-manager-6fc5d867c5-mgfdv/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.703016642Z namespaces/kserve/pods/kserve-controller-manager-6fc5d867c5-mgfdv/manager/manager/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.703081284Z namespaces/kserve/pods/llmisvc-controller-manager-857fbd48cc-6tbhz/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.703105264Z namespaces/kserve/pods/llmisvc-controller-manager-857fbd48cc-6tbhz/llmisvc-controller-manager-857fbd48cc-6tbhz.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.703207867Z namespaces/kserve/pods/llmisvc-controller-manager-857fbd48cc-6tbhz/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.703217037Z namespaces/kserve/pods/llmisvc-controller-manager-857fbd48cc-6tbhz/manager/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.703226357Z namespaces/kserve/pods/llmisvc-controller-manager-857fbd48cc-6tbhz/manager/manager/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.703272989Z namespaces/kserve/pods/llmisvc-controller-manager-857fbd48cc-6tbhz/manager/manager/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.705671357Z namespaces/kserve/pods/llmisvc-controller-manager-857fbd48cc-6tbhz/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.705754099Z namespaces/kserve/pods/llmisvc-controller-manager-857fbd48cc-6tbhz/manager/manager/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.705823151Z namespaces/kserve/pods/model-serving-api-86f7b4b499-tv9wm/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.705875262Z namespaces/kserve/pods/model-serving-api-86f7b4b499-tv9wm/model-serving-api-86f7b4b499-tv9wm.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.706002325Z namespaces/kserve/pods/model-serving-api-86f7b4b499-tv9wm/server/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.706015406Z namespaces/kserve/pods/model-serving-api-86f7b4b499-tv9wm/server/server/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.706028986Z namespaces/kserve/pods/model-serving-api-86f7b4b499-tv9wm/server/server/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.706070307Z namespaces/kserve/pods/model-serving-api-86f7b4b499-tv9wm/server/server/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.70618506Z namespaces/kserve/pods/model-serving-api-86f7b4b499-tv9wm/server/server/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.706280423Z namespaces/kserve/pods/model-serving-api-86f7b4b499-tv9wm/server/server/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.706330054Z namespaces/kserve/pods/odh-model-controller-696fc77849-kptkb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.706381355Z namespaces/kserve/pods/odh-model-controller-696fc77849-kptkb/odh-model-controller-696fc77849-kptkb.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.706456777Z namespaces/kserve/pods/odh-model-controller-696fc77849-kptkb/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.706467117Z namespaces/kserve/pods/odh-model-controller-696fc77849-kptkb/manager/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.706480417Z namespaces/kserve/pods/odh-model-controller-696fc77849-kptkb/manager/manager/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.706517978Z namespaces/kserve/pods/odh-model-controller-696fc77849-kptkb/manager/manager/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.714507545Z namespaces/kserve/pods/odh-model-controller-696fc77849-kptkb/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.714576236Z namespaces/kserve/pods/odh-model-controller-696fc77849-kptkb/manager/manager/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.714615137Z namespaces/kserve/pods/s3-init-jcm8g/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.714657448Z namespaces/kserve/pods/s3-init-jcm8g/s3-init-jcm8g.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.71474037Z namespaces/kserve/pods/s3-init-jcm8g/s3-init/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.714751711Z namespaces/kserve/pods/s3-init-jcm8g/s3-init/s3-init/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.714756631Z namespaces/kserve/pods/s3-init-jcm8g/s3-init/s3-init/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.714774331Z namespaces/kserve/pods/s3-init-jcm8g/s3-init/s3-init/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.714882154Z namespaces/kserve/pods/s3-init-jcm8g/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.714984036Z namespaces/kserve/pods/s3-init-jcm8g/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.715024657Z namespaces/kserve/pods/seaweedfs-86cc847c5c-4wkm2/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.715059518Z namespaces/kserve/pods/seaweedfs-86cc847c5c-4wkm2/seaweedfs-86cc847c5c-4wkm2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.71514969Z namespaces/kserve/pods/seaweedfs-86cc847c5c-4wkm2/seaweedfs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.715158891Z namespaces/kserve/pods/seaweedfs-86cc847c5c-4wkm2/seaweedfs/seaweedfs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.715162691Z namespaces/kserve/pods/seaweedfs-86cc847c5c-4wkm2/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.715192851Z namespaces/kserve/pods/seaweedfs-86cc847c5c-4wkm2/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.715403617Z namespaces/kserve/pods/seaweedfs-86cc847c5c-4wkm2/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.715476209Z namespaces/kserve/pods/seaweedfs-86cc847c5c-4wkm2/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.71553104Z namespaces/kserve/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.715578821Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.715636193Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.715681023Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.715758656Z namespaces/kuadrant-system/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.715810647Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.715895029Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.715973341Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.716039752Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.716084864Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.716181736Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.716479243Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.71675842Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.716807561Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.716852472Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.716927944Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.716989586Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.717080908Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.71715091Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.717196811Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.717280023Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.717322014Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.717330194Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.717378615Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.717467827Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.71757629Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.717658332Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.717754805Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.717792675Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.71797066Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.718104083Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.718510203Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.718592655Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.718926843Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.719060837Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.719268662Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.719404425Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.719441676Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.719554619Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.71958491Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.719635171Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.719674972Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.719763354Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.719845436Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.719896357Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.719959469Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.7200068Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.720053351Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.720116923Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.720126733Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.720177034Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.720373939Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.720596064Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.720746948Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.721005305Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.721304062Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.721543218Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.721570608Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-zqbjk.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.721836535Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.721880646Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.72205111Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.722149083Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.722249955Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.722323497Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.722374448Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.72243812Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.722492311Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.722589713Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.722747507Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.72285846Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.722956093Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.722970513Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.723018074Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/authorino-68bd676465-vlxjh.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.723105426Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/authorino/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.723116396Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/authorino/authorino/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.723130537Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/authorino/authorino/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.723190908Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/authorino/authorino/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724026189Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724098581Z namespaces/kuadrant-system/pods/authorino-68bd676465-vlxjh/authorino/authorino/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724155872Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724207523Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/authorino-operator-7587b89b76-hnn8l.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724302715Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724311426Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/manager/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724315726Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/manager/manager/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724341687Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/manager/manager/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724522731Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724598843Z namespaces/kuadrant-system/pods/authorino-operator-7587b89b76-hnn8l/manager/manager/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724652924Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724703695Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/dns-operator-controller-manager-844548ff4c-hhjsv.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724788578Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724801578Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/manager/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724805538Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/manager/manager/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.724859149Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/manager/manager/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.725012113Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.725085235Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-844548ff4c-hhjsv/manager/manager/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.725139906Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.725192068Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/kuadrant-console-plugin-6c886788f8-mj4k9.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.725274439Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/kuadrant-console-plugin/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.72529033Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.72530163Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.725349991Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.725437003Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.725505485Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-6c886788f8-mj4k9/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.725546016Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.725596597Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/kuadrant-operator-controller-manager-6ddf9554fc-s54pc.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.72569312Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.72570709Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/manager/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.72571156Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/manager/manager/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.725760831Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/manager/manager/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.72691669Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727009882Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6ddf9554fc-s54pc/manager/manager/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727046573Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727093584Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/limitador-limitador-67566c68b4-52nnx.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727193377Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/limitador/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727200707Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/limitador/limitador/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727204717Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/limitador/limitador/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727236918Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/limitador/limitador/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.72733341Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727409462Z namespaces/kuadrant-system/pods/limitador-limitador-67566c68b4-52nnx/limitador/limitador/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727452213Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727489764Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/limitador-operator-controller-manager-c7fb4c8d5-l5476.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727576046Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727587996Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/manager/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727600767Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/manager/manager/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727648248Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/manager/manager/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727833642Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727931005Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-c7fb4c8d5-l5476/manager/manager/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.727968866Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.728005067Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.728063808Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.728115599Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.728174361Z namespaces/kube-node-lease/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.728185301Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.728198551Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.728237022Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-137-68.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.728336615Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-138-68.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.728425477Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-143-218.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.728489969Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.728497479Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.72854242Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.728702154Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.728933549Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.729106004Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.729324179Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.729582105Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.72977775Z namespaces/kube-public/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.729787801Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.729794201Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.729851942Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.730049517Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.730229591Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.730368695Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.73059094Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.730837606Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.731099423Z namespaces/kube-system/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.731148754Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.731225006Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.731270017Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.731344139Z namespaces/kube-system/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.7313788Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.731546024Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.731626286Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.731720358Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.73179988Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.731846151Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.731929873Z namespaces/kube-system/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.731976674Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.732066366Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.732131738Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.7321967Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.732300492Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.732361624Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.732369724Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.732408225Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-a2vefrtg72etmv7tqide74fsva.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.732525898Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.73263386Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.732726193Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.732875776Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.732969209Z namespaces/kube-system/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.73301707Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.733175744Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.733267116Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.733508132Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.733605214Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.733865571Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.734012144Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.734335412Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.734407114Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.734452745Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.734525837Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.734570448Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.734621949Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.734665181Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.734749683Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.734835505Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.734917207Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.735007749Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.735071131Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.735119552Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.735172333Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.735182333Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.735250495Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.735408549Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.735645034Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.735779338Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.736024754Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.73626165Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.736481915Z namespaces/kube-system/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.736497215Z namespaces/kube-system/pods/global-pull-secret-syncer-2kw6v/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.736537307Z namespaces/kube-system/pods/global-pull-secret-syncer-2kw6v/global-pull-secret-syncer-2kw6v.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.736633669Z namespaces/kube-system/pods/global-pull-secret-syncer-2kw6v/global-pull-secret-syncer/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.736643159Z namespaces/kube-system/pods/global-pull-secret-syncer-2kw6v/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.736647019Z namespaces/kube-system/pods/global-pull-secret-syncer-2kw6v/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.73669024Z namespaces/kube-system/pods/global-pull-secret-syncer-2kw6v/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.73707025Z namespaces/kube-system/pods/global-pull-secret-syncer-2kw6v/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.737138741Z namespaces/kube-system/pods/global-pull-secret-syncer-2kw6v/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.737183472Z namespaces/kube-system/pods/global-pull-secret-syncer-hhn8r/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.737227673Z namespaces/kube-system/pods/global-pull-secret-syncer-hhn8r/global-pull-secret-syncer-hhn8r.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.737312366Z namespaces/kube-system/pods/global-pull-secret-syncer-hhn8r/global-pull-secret-syncer/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.737320386Z namespaces/kube-system/pods/global-pull-secret-syncer-hhn8r/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.737331446Z namespaces/kube-system/pods/global-pull-secret-syncer-hhn8r/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.737409078Z namespaces/kube-system/pods/global-pull-secret-syncer-hhn8r/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.737744576Z namespaces/kube-system/pods/global-pull-secret-syncer-hhn8r/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.737814698Z namespaces/kube-system/pods/global-pull-secret-syncer-hhn8r/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.737859909Z namespaces/kube-system/pods/global-pull-secret-syncer-p5w64/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.73791803Z namespaces/kube-system/pods/global-pull-secret-syncer-p5w64/global-pull-secret-syncer-p5w64.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.738060374Z namespaces/kube-system/pods/global-pull-secret-syncer-p5w64/global-pull-secret-syncer/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.738073034Z namespaces/kube-system/pods/global-pull-secret-syncer-p5w64/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.738077134Z namespaces/kube-system/pods/global-pull-secret-syncer-p5w64/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.738107625Z namespaces/kube-system/pods/global-pull-secret-syncer-p5w64/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.738493264Z namespaces/kube-system/pods/global-pull-secret-syncer-p5w64/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.738595977Z namespaces/kube-system/pods/global-pull-secret-syncer-p5w64/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.738628048Z namespaces/kube-system/pods/konnectivity-agent-bxxbs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.738681229Z namespaces/kube-system/pods/konnectivity-agent-bxxbs/konnectivity-agent-bxxbs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.738775431Z namespaces/kube-system/pods/konnectivity-agent-bxxbs/konnectivity-agent/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.738782192Z namespaces/kube-system/pods/konnectivity-agent-bxxbs/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.738789272Z namespaces/kube-system/pods/konnectivity-agent-bxxbs/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.738834533Z namespaces/kube-system/pods/konnectivity-agent-bxxbs/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.739848598Z namespaces/kube-system/pods/konnectivity-agent-bxxbs/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.739962421Z namespaces/kube-system/pods/konnectivity-agent-bxxbs/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.740025832Z namespaces/kube-system/pods/konnectivity-agent-j6hxm/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.740072614Z namespaces/kube-system/pods/konnectivity-agent-j6hxm/konnectivity-agent-j6hxm.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.740151125Z namespaces/kube-system/pods/konnectivity-agent-j6hxm/konnectivity-agent/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.740160706Z namespaces/kube-system/pods/konnectivity-agent-j6hxm/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.740175166Z namespaces/kube-system/pods/konnectivity-agent-j6hxm/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.740216427Z namespaces/kube-system/pods/konnectivity-agent-j6hxm/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.741265393Z namespaces/kube-system/pods/konnectivity-agent-j6hxm/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.741334085Z namespaces/kube-system/pods/konnectivity-agent-j6hxm/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.741387966Z namespaces/kube-system/pods/konnectivity-agent-w6l25/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.741433067Z namespaces/kube-system/pods/konnectivity-agent-w6l25/konnectivity-agent-w6l25.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.741518009Z namespaces/kube-system/pods/konnectivity-agent-w6l25/konnectivity-agent/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.741525009Z namespaces/kube-system/pods/konnectivity-agent-w6l25/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.74153817Z namespaces/kube-system/pods/konnectivity-agent-w6l25/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.741583641Z namespaces/kube-system/pods/konnectivity-agent-w6l25/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.742639626Z namespaces/kube-system/pods/konnectivity-agent-w6l25/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.742714138Z namespaces/kube-system/pods/konnectivity-agent-w6l25/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.742760809Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.742812121Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.742881992Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal/haproxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.742889073Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.742913973Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.742972045Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.743258252Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.743327914Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-68.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.743382115Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-68.ec2.internal/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.743434756Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-68.ec2.internal/kube-apiserver-proxy-ip-10-0-138-68.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.743498828Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-68.ec2.internal/haproxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.743507388Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-68.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.743520908Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-68.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.7435754Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-68.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.743818535Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-68.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.743892247Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-68.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.743952629Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-218.ec2.internal/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.74398846Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-218.ec2.internal/kube-apiserver-proxy-ip-10-0-143-218.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.744065452Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-218.ec2.internal/haproxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.744077452Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-218.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.744085392Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-218.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.744120353Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-218.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.74439206Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-218.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.744464142Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-218.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.744529923Z namespaces/kube-system/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.744565394Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.744627035Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.744642316Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.744666327Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.744731648Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.744774259Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.74482442Z namespaces/open-cluster-management-121837d2-f4fb-4805-b181-429475e54/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.74483472Z namespaces/open-cluster-management-121837d2-f4fb-4805-b181-429475e54/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.744838811Z namespaces/open-cluster-management-121837d2-f4fb-4805-b181-429475e54/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.744875522Z namespaces/open-cluster-management-121837d2-f4fb-4805-b181-429475e54/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.745069196Z namespaces/open-cluster-management-121837d2-f4fb-4805-b181-429475e54/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.745258151Z namespaces/open-cluster-management-121837d2-f4fb-4805-b181-429475e54/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.745406605Z namespaces/open-cluster-management-121837d2-f4fb-4805-b181-429475e54/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.74563965Z namespaces/open-cluster-management-121837d2-f4fb-4805-b181-429475e54/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.745922237Z namespaces/open-cluster-management-121837d2-f4fb-4805-b181-429475e54/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.746278636Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.746289136Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.746296967Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.746335808Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.74643158Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.746483121Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.746493861Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.746557283Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.746725447Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.746934952Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.747084876Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.747290921Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.747530887Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.747766903Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.747830204Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.747923027Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.747989048Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.74806891Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.748090491Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.748208494Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.748288945Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.748381338Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.748446999Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.74849617Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.748574652Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.748628154Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.748716576Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.748769547Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.748809638Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.74889417Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.749006293Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.749049864Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.749956046Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.75009261Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.750176122Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.750257484Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.750344996Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.750466639Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.750642173Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.750705555Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.750749956Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.750809247Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.750846028Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.75091551Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.750962711Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.751063954Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.751143576Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.751192917Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.751241328Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.75129903Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.751346121Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.751400322Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.751447773Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.751491514Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.751509245Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.751570506Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.751786782Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.751989936Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.75213101Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.752333275Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.752584851Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.752788296Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.752819967Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.752892709Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.75295205Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.753058703Z namespaces/openshift-apiserver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.753108114Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.753170185Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.753209886Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.753275478Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.753323639Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.753412471Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.753492993Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.753585626Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.753638797Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.753684058Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.75374835Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.753797281Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.753878473Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.754017816Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.754063947Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.75416757Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.754253042Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.754311434Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.754428336Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.754514829Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.754600401Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.754681833Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.754772105Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.754897028Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.755081523Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.755139024Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.755187085Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.755251477Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.755296168Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.755358419Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.75540376Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.755494503Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.755574005Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.755727898Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.75577714Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.755846551Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.755890032Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.755964084Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.755982645Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.756026606Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.75618737Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.756378915Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.756527228Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.756721553Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.75699205Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.757201985Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.757237825Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.757305457Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.757336458Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.75740144Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.75741279Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.75742218Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.757455561Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.757523683Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.757538533Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.757591214Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.757756748Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.757960523Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.758102477Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.758311702Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.758554398Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.758769773Z namespaces/openshift-authentication/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.758778234Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.758784763Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.758824775Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.759009279Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.759194994Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.759335497Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.759534492Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.759803459Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.760042905Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.760059815Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.760066965Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.760078615Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.760148927Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.760165857Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.760221519Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.760373813Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.760568878Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.760729311Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.761002478Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.761258884Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.76146742Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.76147867Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.76148328Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.761522621Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.761601993Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.761613663Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.761641594Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.761696215Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.761724716Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.761778807Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.762000393Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.762191378Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.762331571Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.762531056Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.762767412Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.763018708Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.763068239Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.763140451Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.763175632Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.763246393Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.763280954Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.763374976Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.763449618Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.76353454Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.763586852Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.763634663Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.763694214Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.763744425Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.763824098Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.763881329Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.76394869Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.764036573Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.764085304Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.764095644Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.764152486Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.764220617Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.764270169Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.764371041Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.764452313Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.764550456Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.764636118Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.764708519Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.764874013Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765024217Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765087339Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.76513893Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765193741Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765240922Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765286783Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765329824Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765395306Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765444657Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.76552786Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765614211Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765669393Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765726414Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765767065Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765815616Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765862618Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.765895729Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.76596194Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.766156435Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.76637543Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.766527354Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.76675033Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767022136Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767215621Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767261642Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767329364Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767341384Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767360845Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767421546Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767456687Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767511368Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767545149Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767635061Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767681722Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767764895Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767816856Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767874867Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.767929608Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.768066242Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.768156104Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.768241126Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.768287227Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.768325718Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.76837795Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.76841912Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.768508743Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.768563194Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.768619285Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.768706758Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.768759729Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.76879744Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.768837631Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.768941523Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.769055556Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.769169889Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.769271552Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.769308922Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.769418875Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.769505077Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.769837575Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.770002659Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.770217095Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.770408849Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.770533232Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.770584914Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.770640335Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.770701797Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.770752068Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.770810319Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.770867651Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.770941782Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.770982854Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.771069376Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.771153798Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.771203069Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.77125502Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.771299951Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.771339982Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.771413684Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.771423734Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.771462865Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.771618189Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.771801064Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.771980278Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.772216144Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.77246994Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.772848989Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.77286499Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.772923381Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/aws-ebs-csi-driver-node-5bkht.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773052954Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-driver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773063545Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-driver/csi-driver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773083975Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-driver/csi-driver/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773140417Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773239219Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773312261Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773350332Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-liveness-probe/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773357322Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773364102Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773425234Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773543677Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773623748Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.77366262Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-node-driver-registrar/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.77367251Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.77367698Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773712011Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773809883Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773881175Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5bkht/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.773955817Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774010588Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/aws-ebs-csi-driver-node-9v7r4.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774109631Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-driver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774118591Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-driver/csi-driver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774122281Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-driver/csi-driver/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774161112Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774264694Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774342076Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774374287Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-liveness-probe/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774390207Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774397478Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774449879Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774544861Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774613563Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774664214Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-node-driver-registrar/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774672534Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774678754Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774721095Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774826168Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.77491787Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9v7r4/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.774973502Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775034733Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/aws-ebs-csi-driver-node-tr4s7.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775130836Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-driver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775152606Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-driver/csi-driver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775159416Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-driver/csi-driver/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775166666Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775287569Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775348531Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775400792Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-liveness-probe/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775409523Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775419503Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775460134Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775559826Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775627958Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775688689Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-node-driver-registrar/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775696879Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.77570512Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775746301Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775853233Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775957866Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-tr4s7/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.775991317Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776041098Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776093319Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.7761021Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.7761413Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776207252Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776251063Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776323955Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776366466Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776432308Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776439898Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776443348Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776481839Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776552071Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776560131Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776605412Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776677444Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776700204Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.776751576Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.77693812Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.777132435Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.777274238Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.777479903Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.777722839Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.778056588Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.77814027Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.778231612Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.778281703Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.778372525Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.778436397Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.778618041Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.778744954Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.778865047Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.77896216Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.779009061Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.779111194Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.779178265Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.779248007Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.779305058Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.779398481Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.779467682Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.779530174Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.779641296Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.779708718Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.779715568Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.77976613Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.779849812Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.779895493Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.780819865Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78100117Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.781170634Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.781262516Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.781458701Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.781600925Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.781847001Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.781959844Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782039226Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782082887Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782139708Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78219797Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782253251Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782334703Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782426555Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782497757Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782546908Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78262412Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782670951Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782742263Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782789214Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782845085Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782891067Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782963958Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.782980969Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78303591Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.783207244Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.783396439Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.783561303Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78382779Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.784158818Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.784419074Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.784430494Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ltb6d/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.784481726Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ltb6d/tuned-ltb6d.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.784584298Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ltb6d/tuned/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.784595038Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ltb6d/tuned/tuned/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.784604138Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ltb6d/tuned/tuned/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78464992Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ltb6d/tuned/tuned/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.784771713Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ltb6d/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.784849865Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ltb6d/tuned/tuned/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.784914686Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrrqf/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.784979608Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrrqf/tuned-rrrqf.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78507272Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrrqf/tuned/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78508378Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrrqf/tuned/tuned/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.785090101Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrrqf/tuned/tuned/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.785141042Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrrqf/tuned/tuned/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.785259995Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrrqf/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.785370627Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rrrqf/tuned/tuned/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.785426839Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tfs7c/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78549792Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tfs7c/tuned-tfs7c.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.785598873Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tfs7c/tuned/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.785605863Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tfs7c/tuned/tuned/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.785653274Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tfs7c/tuned/tuned/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.785712046Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tfs7c/tuned/tuned/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.785836219Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tfs7c/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.785980442Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tfs7c/tuned/tuned/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.786063484Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.786113996Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.786175177Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.786211168Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78627157Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78628054Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78630752Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-137-68.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.786406643Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-138-68.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.786501625Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-143-218.ec2.internal.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.786555206Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.786598998Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78667828Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.786732261Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.786797252Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.786840333Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.786918315Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.786970997Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.787064149Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.787179052Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.787288525Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.787336846Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.787389877Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.787430598Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.787472539Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.787554971Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.787613823Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.787664244Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.787748276Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.787822758Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.787870039Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.788077844Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.788169586Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78830332Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.788386722Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.788512375Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.788639628Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.788817652Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.788884624Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.788941705Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.789024857Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.789074359Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78914038Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.789186041Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.789269013Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.789354565Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.789432317Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.789475588Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78953366Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.789578331Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.789654863Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.789696184Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.789759515Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.789800756Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.789849478Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.789865568Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.78994844Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.790101964Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.790290358Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.790436502Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.790686808Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.790970985Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.79116902Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.79117573Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6dc5bdb6b4-vdnk8/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.791217791Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6dc5bdb6b4-vdnk8/cluster-samples-operator-6dc5bdb6b4-vdnk8.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.791302493Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6dc5bdb6b4-vdnk8/cluster-samples-operator-watch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.791309263Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6dc5bdb6b4-vdnk8/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.791313074Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6dc5bdb6b4-vdnk8/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.791353464Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6dc5bdb6b4-vdnk8/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.791448417Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6dc5bdb6b4-vdnk8/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.791525039Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6dc5bdb6b4-vdnk8/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.79156406Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6dc5bdb6b4-vdnk8/cluster-samples-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.79157024Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6dc5bdb6b4-vdnk8/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.7915739Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6dc5bdb6b4-vdnk8/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.791617461Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6dc5bdb6b4-vdnk8/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.792325159Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6dc5bdb6b4-vdnk8/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.79240386Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6dc5bdb6b4-vdnk8/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.792442331Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.792479782Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.792539344Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.792592335Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.792681747Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.792734489Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.79280308Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.792847111Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.792928793Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.792987595Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.793078737Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.79318558Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.793292162Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.793407195Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.793449676Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.793517198Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.793560529Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.793651411Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.793701172Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.793766084Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.793835765Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.793885117Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.793891927Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.793967759Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.794056701Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.794128093Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.794192544Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.794288627Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.794376829Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.794669696Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.794753178Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.794868121Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.795021215Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.795274201Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.795351313Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.795401624Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.795464346Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.795505367Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.795566958Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.795610499Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.795707172Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.795784944Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.795853645Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.795918737Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.795977518Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.79602851Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.796113862Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.796160173Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.796225735Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.796268516Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.796349087Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.796365738Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.796414649Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.796571783Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.796817309Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.796996683Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.797230009Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.797479505Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.79767539Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.79768911Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-7c6cbb6c87-qgzxg/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.797735572Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-7c6cbb6c87-qgzxg/volume-data-source-validator-7c6cbb6c87-qgzxg.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.797838204Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-7c6cbb6c87-qgzxg/volume-data-source-validator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.797850154Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-7c6cbb6c87-qgzxg/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.797860665Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-7c6cbb6c87-qgzxg/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.797912486Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-7c6cbb6c87-qgzxg/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.798029859Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-7c6cbb6c87-qgzxg/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.798105491Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-7c6cbb6c87-qgzxg/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.798164452Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.798203263Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.798271785Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.798322006Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.798558372Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.798609393Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.798681425Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.798726146Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.798785067Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.798830408Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.798939741Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.799080374Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.799163307Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.799218708Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.799270839Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.799328211Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.799379512Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.799466184Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.799525316Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.799570357Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.799655429Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.79971332Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.79972055Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.799766211Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.799820533Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.799871604Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.799999927Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.800083559Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.800189222Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.800275254Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.800362886Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.800499889Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.800707355Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.800770446Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.800818187Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.800876419Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.800948791Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.801025713Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.801070254Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.801156986Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.801241408Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.801309369Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.80134815Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.801407072Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.801453383Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.801522315Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.801532875Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.801579016Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.80174506Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.801960005Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.802111399Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.802301864Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.80253653Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.802748425Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.802794096Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.802855238Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.802914069Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803037632Z namespaces/openshift-config-managed/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803088463Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803153735Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803192446Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803258807Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803300508Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803392871Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803478173Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803566465Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803615346Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803667977Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803728279Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.80377956Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803864532Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803926604Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.803980405Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.804068067Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.804151879Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.80419558Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.807203754Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.807288387Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.807398039Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.807472581Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.807568333Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.807753558Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.807870221Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.807956683Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808007854Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808105916Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808156548Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808205429Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.80826241Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808306932Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808367453Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808419234Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808505276Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808630989Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808711661Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808762183Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808819514Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808859665Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808925107Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.808999098Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.80905325Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.809221964Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.809408499Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.809574623Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.8098762Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.810186578Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.810438894Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.810482315Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.810549757Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.810559547Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.810596668Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.810667919Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.810709901Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.810771852Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.810807813Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.810861264Z namespaces/openshift-config-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.810869144Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.810878835Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.810947586Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.811012738Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.811026428Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.81108599Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.811238704Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.811419508Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.811570932Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.811801367Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.812076014Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.81231376Z namespaces/openshift-config/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.812359871Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.812524495Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.812571046Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.812640868Z namespaces/openshift-config/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.812687969Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.812777282Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.812864714Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.812974476Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.813049708Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.813098749Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.813169611Z namespaces/openshift-config/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.813231433Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.813326325Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.813377686Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.813419937Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.813505579Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.813595902Z namespaces/openshift-config/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.813651173Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.813801967Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.813880009Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.813992091Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.814077673Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.814159245Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.814312319Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.814453763Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.814515224Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.814566986Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.814625907Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.814675868Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.81474185Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.814795611Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.814850802Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.814914814Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.815010026Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.815088258Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.815138929Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.815182991Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.815231892Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.815277883Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.815324674Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.815330804Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.815379715Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.815527839Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.815711434Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.815849037Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.816077953Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.816316409Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.816522454Z namespaces/openshift-config/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.816564595Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.816622376Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.816660777Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.816745419Z namespaces/openshift-console-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.81679578Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.816864412Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.816927733Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.817075707Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.817122838Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.817213791Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.817335274Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.817444246Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.817490937Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.817541448Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.81760149Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.817641291Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.817734463Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.817885047Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.817957229Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.818050691Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.818102502Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.818110733Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.818149114Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.818227176Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.818280157Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.818862411Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.818974684Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.819319142Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.819407034Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.819547038Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.819689992Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.819913367Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.819993189Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.82004145Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.820099182Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.820141723Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.820195554Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.820236845Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.820327167Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.820408209Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.82045474Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.820499621Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.820557643Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.820609294Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.820662295Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.820719487Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.820766778Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.820811969Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.82086117Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.82087305Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.821040745Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.821194358Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.821376783Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.821516916Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.821717641Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.821995448Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.822203283Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.822211283Z namespaces/openshift-console-operator/pods/console-operator-9d4b6777b-5nzks/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.822257804Z namespaces/openshift-console-operator/pods/console-operator-9d4b6777b-5nzks/console-operator-9d4b6777b-5nzks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.822347357Z namespaces/openshift-console-operator/pods/console-operator-9d4b6777b-5nzks/console-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.822358047Z namespaces/openshift-console-operator/pods/console-operator-9d4b6777b-5nzks/console-operator/console-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.822364377Z namespaces/openshift-console-operator/pods/console-operator-9d4b6777b-5nzks/console-operator/console-operator/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.822406938Z namespaces/openshift-console-operator/pods/console-operator-9d4b6777b-5nzks/console-operator/console-operator/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.823185238Z namespaces/openshift-console-operator/pods/console-operator-9d4b6777b-5nzks/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.823234699Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.82328199Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.823326891Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.823383972Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.823432044Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.823439904Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.823449664Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.823506565Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.823661639Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.823852214Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.824026868Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.824231623Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.824474229Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.824718105Z namespaces/openshift-console/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.824768526Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.824999502Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.825057224Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.825130165Z namespaces/openshift-console/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.825181396Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.825269149Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.825441543Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.825699579Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.825746961Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.825791391Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.825846193Z namespaces/openshift-console/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.825885654Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.826001917Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.826043688Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.826095369Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.826170661Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.826238133Z namespaces/openshift-console/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.826287784Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.826884558Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.826998181Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.827271828Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.82736342Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.827532194Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.827719019Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.828067247Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.828136779Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.82817652Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.828244542Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.828287733Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.828353485Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.828393186Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.828482648Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.82858523Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.828639872Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.828692183Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.828754715Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.828805606Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.828859247Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.828916178Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.82899979Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.829012811Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.829050242Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.829211036Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.82939694Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.829541864Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.82977704Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830055556Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830271782Z namespaces/openshift-console/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830278262Z namespaces/openshift-console/pods/console-5b498c6d49-nlgtd/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830322963Z namespaces/openshift-console/pods/console-5b498c6d49-nlgtd/console-5b498c6d49-nlgtd.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830424736Z namespaces/openshift-console/pods/console-5b498c6d49-nlgtd/console/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830431376Z namespaces/openshift-console/pods/console-5b498c6d49-nlgtd/console/console/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830436976Z namespaces/openshift-console/pods/console-5b498c6d49-nlgtd/console/console/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830489277Z namespaces/openshift-console/pods/console-5b498c6d49-nlgtd/console/console/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.83060033Z namespaces/openshift-console/pods/console-5b498c6d49-nlgtd/console/console/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830675022Z namespaces/openshift-console/pods/console-5b498c6d49-nlgtd/console/console/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830734553Z namespaces/openshift-console/pods/downloads-6bcc868b7-b85zj/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830776184Z namespaces/openshift-console/pods/downloads-6bcc868b7-b85zj/downloads-6bcc868b7-b85zj.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830871176Z namespaces/openshift-console/pods/downloads-6bcc868b7-b85zj/download-server/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830879257Z namespaces/openshift-console/pods/downloads-6bcc868b7-b85zj/download-server/download-server/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830883557Z namespaces/openshift-console/pods/downloads-6bcc868b7-b85zj/download-server/download-server/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.830951058Z namespaces/openshift-console/pods/downloads-6bcc868b7-b85zj/download-server/download-server/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.831321898Z namespaces/openshift-console/pods/downloads-6bcc868b7-b85zj/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.831396439Z namespaces/openshift-console/pods/downloads-6bcc868b7-b85zj/download-server/download-server/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.831449631Z namespaces/openshift-console/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.831494752Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.831577974Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.831621145Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.831689487Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.831733408Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.831909992Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.832006754Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.832068816Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.832119617Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.832205389Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.832289451Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.832374803Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.832432245Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.832483576Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.832539528Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.832590839Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.832693411Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.832754643Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.832823215Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.832965618Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.8330414Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.833087021Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.833207564Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.833288986Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.833370668Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.83346226Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.833547392Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.833677205Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.83385382Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.833939322Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.834007714Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.834070135Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.834121356Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.834187598Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.834236059Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.834327401Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.834413303Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.834458065Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.834499516Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.834542957Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.834593758Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.834651559Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.83466416Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.834719391Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.834925726Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.835135451Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.835284325Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.83550097Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.835737976Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.835959241Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.836006083Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.836062014Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.836104535Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.836181887Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.836232748Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.83629537Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.836338541Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.836426423Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.836463894Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.836558266Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.836641108Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.83672596Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.836773432Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.836823863Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.836879394Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.836964306Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.837063119Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.83714051Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.837190382Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.837281744Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.837324385Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.837335505Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.837392087Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.837446038Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.837498289Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.837613222Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.837694284Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.837786037Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.837869019Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.838020682Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.838174856Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.838388531Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.838519895Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.838564256Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.838630657Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.838675318Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.83873113Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.838780091Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.838870223Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.838971156Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.839027297Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.839078048Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.83913461Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.839184031Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.839243972Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.839251493Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.839307534Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.839463078Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.839644682Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.839783175Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.840015691Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.840254877Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.840475263Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.840521964Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.840591506Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.840636027Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.840699238Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.840747819Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.840866422Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.840931684Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.841026846Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.841076147Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.84118633Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.841289043Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.841390815Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.841456097Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.841512178Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.84159398Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.841650581Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.841752924Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.841825706Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.841885937Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.842017121Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.842083352Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.842137824Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.842268737Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.842374209Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.842480232Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.842584975Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.842679917Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.842844631Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.843121228Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.84319945Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.843256931Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.843340963Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.843395164Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.843464236Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.843520978Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.84361226Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.843696242Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.843766404Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.843815915Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.843870966Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.843946648Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.84401163Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.844065821Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.844117392Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.844164853Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.844214315Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.844238445Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.844291877Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.84444642Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.844639155Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.844782189Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.845043465Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.845376843Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.8456499Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.845756442Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.845813214Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.845875876Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.846013979Z namespaces/openshift-dns/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.846085301Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.846679015Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.846738517Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.846820019Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.84686035Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.847051604Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.847135076Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.847226339Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.84727487Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.847321141Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.847383533Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.847438674Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.847534306Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.847594028Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.847660989Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.847772682Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.847844064Z namespaces/openshift-dns/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.847918536Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.848048659Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.848147951Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.848442368Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.848539001Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.848826348Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.849000842Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.849185567Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.849255988Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.84929955Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.849375882Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.849399152Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.849470934Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.849517285Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.849608177Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.849689979Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.849754841Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.849803572Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.849856113Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.849923395Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.849977656Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.850024067Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.850073819Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.850096499Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.850152141Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.850316885Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.850502599Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.850646383Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.850853418Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.851125794Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85135353Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85136324Z namespaces/openshift-dns/pods/dns-default-9786k/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.851399641Z namespaces/openshift-dns/pods/dns-default-9786k/dns-default-9786k.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.851493774Z namespaces/openshift-dns/pods/dns-default-9786k/dns/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.851500794Z namespaces/openshift-dns/pods/dns-default-9786k/dns/dns/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.851504754Z namespaces/openshift-dns/pods/dns-default-9786k/dns/dns/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.851542355Z namespaces/openshift-dns/pods/dns-default-9786k/dns/dns/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.851643367Z namespaces/openshift-dns/pods/dns-default-9786k/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.851717049Z namespaces/openshift-dns/pods/dns-default-9786k/dns/dns/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85175823Z namespaces/openshift-dns/pods/dns-default-9786k/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85176562Z namespaces/openshift-dns/pods/dns-default-9786k/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85176931Z namespaces/openshift-dns/pods/dns-default-9786k/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.851813502Z namespaces/openshift-dns/pods/dns-default-9786k/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.851917134Z namespaces/openshift-dns/pods/dns-default-9786k/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852009026Z namespaces/openshift-dns/pods/dns-default-9786k/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852059668Z namespaces/openshift-dns/pods/dns-default-fxb9d/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852107079Z namespaces/openshift-dns/pods/dns-default-fxb9d/dns-default-fxb9d.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852194451Z namespaces/openshift-dns/pods/dns-default-fxb9d/dns/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852201171Z namespaces/openshift-dns/pods/dns-default-fxb9d/dns/dns/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852206791Z namespaces/openshift-dns/pods/dns-default-fxb9d/dns/dns/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852258213Z namespaces/openshift-dns/pods/dns-default-fxb9d/dns/dns/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852361185Z namespaces/openshift-dns/pods/dns-default-fxb9d/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852436727Z namespaces/openshift-dns/pods/dns-default-fxb9d/dns/dns/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852494498Z namespaces/openshift-dns/pods/dns-default-fxb9d/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852505399Z namespaces/openshift-dns/pods/dns-default-fxb9d/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852510539Z namespaces/openshift-dns/pods/dns-default-fxb9d/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85255084Z namespaces/openshift-dns/pods/dns-default-fxb9d/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852654632Z namespaces/openshift-dns/pods/dns-default-fxb9d/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852727994Z namespaces/openshift-dns/pods/dns-default-fxb9d/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852766145Z namespaces/openshift-dns/pods/dns-default-lg22b/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852813766Z namespaces/openshift-dns/pods/dns-default-lg22b/dns-default-lg22b.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852894358Z namespaces/openshift-dns/pods/dns-default-lg22b/dns/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852920739Z namespaces/openshift-dns/pods/dns-default-lg22b/dns/dns/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.852929869Z namespaces/openshift-dns/pods/dns-default-lg22b/dns/dns/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85298244Z namespaces/openshift-dns/pods/dns-default-lg22b/dns/dns/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853080582Z namespaces/openshift-dns/pods/dns-default-lg22b/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853157315Z namespaces/openshift-dns/pods/dns-default-lg22b/dns/dns/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853212906Z namespaces/openshift-dns/pods/dns-default-lg22b/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853222886Z namespaces/openshift-dns/pods/dns-default-lg22b/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853230996Z namespaces/openshift-dns/pods/dns-default-lg22b/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853274007Z namespaces/openshift-dns/pods/dns-default-lg22b/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85337084Z namespaces/openshift-dns/pods/dns-default-lg22b/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853452872Z namespaces/openshift-dns/pods/dns-default-lg22b/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853503473Z namespaces/openshift-dns/pods/node-resolver-728bd/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853558404Z namespaces/openshift-dns/pods/node-resolver-728bd/node-resolver-728bd.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853641176Z namespaces/openshift-dns/pods/node-resolver-728bd/dns-node-resolver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853650587Z namespaces/openshift-dns/pods/node-resolver-728bd/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853654137Z namespaces/openshift-dns/pods/node-resolver-728bd/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853692278Z namespaces/openshift-dns/pods/node-resolver-728bd/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85378839Z namespaces/openshift-dns/pods/node-resolver-728bd/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853869732Z namespaces/openshift-dns/pods/node-resolver-728bd/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853943404Z namespaces/openshift-dns/pods/node-resolver-mtx92/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.853979215Z namespaces/openshift-dns/pods/node-resolver-mtx92/node-resolver-mtx92.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854082937Z namespaces/openshift-dns/pods/node-resolver-mtx92/dns-node-resolver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854092827Z namespaces/openshift-dns/pods/node-resolver-mtx92/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854096838Z namespaces/openshift-dns/pods/node-resolver-mtx92/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854140409Z namespaces/openshift-dns/pods/node-resolver-mtx92/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854237741Z namespaces/openshift-dns/pods/node-resolver-mtx92/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854306843Z namespaces/openshift-dns/pods/node-resolver-mtx92/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854352944Z namespaces/openshift-dns/pods/node-resolver-v6skv/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854404895Z namespaces/openshift-dns/pods/node-resolver-v6skv/node-resolver-v6skv.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854473277Z namespaces/openshift-dns/pods/node-resolver-v6skv/dns-node-resolver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854489307Z namespaces/openshift-dns/pods/node-resolver-v6skv/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854499968Z namespaces/openshift-dns/pods/node-resolver-v6skv/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854546219Z namespaces/openshift-dns/pods/node-resolver-v6skv/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854641831Z namespaces/openshift-dns/pods/node-resolver-v6skv/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854720203Z namespaces/openshift-dns/pods/node-resolver-v6skv/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854747174Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854796565Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854855906Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.854922688Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855066751Z namespaces/openshift-etcd/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855114583Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855176364Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855227185Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855289367Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855336258Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85542862Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855509012Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855590914Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855638916Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855691607Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855732658Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855783559Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855865031Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855945663Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.855989744Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.856081276Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.856153388Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85620635Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.856308142Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.856398404Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.856483076Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.856568319Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85664483Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.856774564Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.856974829Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85702824Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.857065721Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.857123302Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.857156053Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.857211064Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.857256785Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.857349148Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85743144Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.857475551Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.857523032Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.857574703Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.857628894Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.857687146Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.857699506Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.857746777Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.857929572Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.858124717Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85826767Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.858522167Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.858778953Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859012919Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859043939Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859116921Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859152242Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859238504Z namespaces/openshift-host-network/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859287905Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859365467Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859413308Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.85948044Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859509471Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859598093Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859679515Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859769747Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859820718Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.8598715Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859958512Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.859999443Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.860088095Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.860136746Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.860180877Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.860262569Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.860343551Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.860384962Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.860489615Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.860570507Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.860653899Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.860739751Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.860826973Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.861012578Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.86112176Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.861179932Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.861225973Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.861290195Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.861338036Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.861399647Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.861444478Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.86150452Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.861552761Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.861635903Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.861715875Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.861774036Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.861823328Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.861882649Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.861948751Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.862012232Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.862018522Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.862074124Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.862233338Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.862415362Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.862555276Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.862752311Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.863079359Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.863302994Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.863338975Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.863398896Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.863426947Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.8635477Z namespaces/openshift-image-registry/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.863597011Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.863679753Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.863722414Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.863787356Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.863829567Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.864022082Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.864174215Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.864377711Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.864425821Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.864475143Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.864522534Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.864570455Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.864675828Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.864742819Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.864797301Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.864888353Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.864970525Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.864981415Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.865025736Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.865100318Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.865126769Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.866029251Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.866138694Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.866432091Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.866507633Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.866719328Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.866861592Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.8671902Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.867246891Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.867292142Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.867348153Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.867395265Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.867447756Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.867506907Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.86761996Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.867739403Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.867808925Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.867855536Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.867950068Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.868002889Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.868112582Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.868206994Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.868258036Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.868309367Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.868362169Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.86841342Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.868465191Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.868476801Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.868523682Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.868712797Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.868977484Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.869148368Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.869376123Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.86962694Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.869839705Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.869847535Z namespaces/openshift-image-registry/pods/image-registry-6f99d9d489-gqwc4/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.869920797Z namespaces/openshift-image-registry/pods/image-registry-6f99d9d489-gqwc4/image-registry-6f99d9d489-gqwc4.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.870030219Z namespaces/openshift-image-registry/pods/image-registry-6f99d9d489-gqwc4/registry/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.870038179Z namespaces/openshift-image-registry/pods/image-registry-6f99d9d489-gqwc4/registry/registry/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.87004135Z namespaces/openshift-image-registry/pods/image-registry-6f99d9d489-gqwc4/registry/registry/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.870088951Z namespaces/openshift-image-registry/pods/image-registry-6f99d9d489-gqwc4/registry/registry/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872038049Z namespaces/openshift-image-registry/pods/image-registry-6f99d9d489-gqwc4/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872116651Z namespaces/openshift-image-registry/pods/image-registry-6f99d9d489-gqwc4/registry/registry/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872134601Z namespaces/openshift-image-registry/pods/node-ca-4f55z/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872192323Z namespaces/openshift-image-registry/pods/node-ca-4f55z/node-ca-4f55z.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872283865Z namespaces/openshift-image-registry/pods/node-ca-4f55z/node-ca/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872292575Z namespaces/openshift-image-registry/pods/node-ca-4f55z/node-ca/node-ca/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872299845Z namespaces/openshift-image-registry/pods/node-ca-4f55z/node-ca/node-ca/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872346976Z namespaces/openshift-image-registry/pods/node-ca-4f55z/node-ca/node-ca/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872566742Z namespaces/openshift-image-registry/pods/node-ca-4f55z/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872644304Z namespaces/openshift-image-registry/pods/node-ca-4f55z/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872681955Z namespaces/openshift-image-registry/pods/node-ca-6lc2b/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872728586Z namespaces/openshift-image-registry/pods/node-ca-6lc2b/node-ca-6lc2b.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872801107Z namespaces/openshift-image-registry/pods/node-ca-6lc2b/node-ca/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872809318Z namespaces/openshift-image-registry/pods/node-ca-6lc2b/node-ca/node-ca/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872813458Z namespaces/openshift-image-registry/pods/node-ca-6lc2b/node-ca/node-ca/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.872857329Z namespaces/openshift-image-registry/pods/node-ca-6lc2b/node-ca/node-ca/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873063634Z namespaces/openshift-image-registry/pods/node-ca-6lc2b/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873124926Z namespaces/openshift-image-registry/pods/node-ca-6lc2b/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873185287Z namespaces/openshift-image-registry/pods/node-ca-xlkpg/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873227518Z namespaces/openshift-image-registry/pods/node-ca-xlkpg/node-ca-xlkpg.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.87329622Z namespaces/openshift-image-registry/pods/node-ca-xlkpg/node-ca/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.87330257Z namespaces/openshift-image-registry/pods/node-ca-xlkpg/node-ca/node-ca/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.87330617Z namespaces/openshift-image-registry/pods/node-ca-xlkpg/node-ca/node-ca/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873354321Z namespaces/openshift-image-registry/pods/node-ca-xlkpg/node-ca/node-ca/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873534636Z namespaces/openshift-image-registry/pods/node-ca-xlkpg/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873604397Z namespaces/openshift-image-registry/pods/node-ca-xlkpg/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873654139Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873681339Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873753361Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873789992Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873843823Z namespaces/openshift-infra/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873849793Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873856993Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.873930955Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.87410313Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.874281894Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.874419487Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.874625092Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.874863478Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.875164406Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.875216547Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.875282239Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.87533097Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.875382471Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.875433212Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.875546915Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.875629007Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.875716459Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.87576016Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.875805692Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.875854553Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.875913194Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.876005606Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.876056438Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.876097429Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.876177791Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.876253283Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.876301314Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.876435957Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.87656398Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.876726024Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.876818166Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.877004421Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.877133164Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.877308249Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.877432281Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.877479453Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.877541314Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.877584965Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.877648457Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.877691658Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.87777997Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.877859402Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.877938064Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.877985505Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.878039546Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.878079297Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.878132559Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.878149679Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.8782063Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.878358264Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.878542209Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.878684342Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.878894168Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.879175824Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.87939715Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.87940776Z namespaces/openshift-ingress-canary/pods/ingress-canary-bwrpd/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.879464102Z namespaces/openshift-ingress-canary/pods/ingress-canary-bwrpd/ingress-canary-bwrpd.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.879955034Z namespaces/openshift-ingress-canary/pods/ingress-canary-bwrpd/serve-healthcheck-canary/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.879966654Z namespaces/openshift-ingress-canary/pods/ingress-canary-bwrpd/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.879981894Z namespaces/openshift-ingress-canary/pods/ingress-canary-bwrpd/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.880030836Z namespaces/openshift-ingress-canary/pods/ingress-canary-bwrpd/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.880158999Z namespaces/openshift-ingress-canary/pods/ingress-canary-bwrpd/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.880251111Z namespaces/openshift-ingress-canary/pods/ingress-canary-bwrpd/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.880318983Z namespaces/openshift-ingress-canary/pods/ingress-canary-jw7bs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.880371004Z namespaces/openshift-ingress-canary/pods/ingress-canary-jw7bs/ingress-canary-jw7bs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.880462356Z namespaces/openshift-ingress-canary/pods/ingress-canary-jw7bs/serve-healthcheck-canary/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.880469086Z namespaces/openshift-ingress-canary/pods/ingress-canary-jw7bs/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.880474616Z namespaces/openshift-ingress-canary/pods/ingress-canary-jw7bs/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.880544538Z namespaces/openshift-ingress-canary/pods/ingress-canary-jw7bs/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.880664041Z namespaces/openshift-ingress-canary/pods/ingress-canary-jw7bs/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.880761353Z namespaces/openshift-ingress-canary/pods/ingress-canary-jw7bs/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.880819885Z namespaces/openshift-ingress-canary/pods/ingress-canary-z8pdq/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.880877266Z namespaces/openshift-ingress-canary/pods/ingress-canary-z8pdq/ingress-canary-z8pdq.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.880994359Z namespaces/openshift-ingress-canary/pods/ingress-canary-z8pdq/serve-healthcheck-canary/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.881005229Z namespaces/openshift-ingress-canary/pods/ingress-canary-z8pdq/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.881015909Z namespaces/openshift-ingress-canary/pods/ingress-canary-z8pdq/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.881070391Z namespaces/openshift-ingress-canary/pods/ingress-canary-z8pdq/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.881185934Z namespaces/openshift-ingress-canary/pods/ingress-canary-z8pdq/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.881290836Z namespaces/openshift-ingress-canary/pods/ingress-canary-z8pdq/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.881330607Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.881390739Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.88145409Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.881505942Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.881618284Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.881668216Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.881742258Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.881789769Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.88185811Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.881926702Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.882029294Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.882127487Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.882220909Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.882280301Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.882337232Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.882399344Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.882460175Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.882567148Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.8826432Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.882707151Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.882821504Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.882896456Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.882972468Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.883810468Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.883916511Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.884093465Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.884183847Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.88427277Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.884402073Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.884590108Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.884652469Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.88470204Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.884755932Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.884801493Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.884860434Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.884867474Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.884936326Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.885045009Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.88509676Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.885233353Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.885322486Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.885398977Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.885439338Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.88550896Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.885570502Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.885656314Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.885701905Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.885770277Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.885815958Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.885870199Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.885876389Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.88593195Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.886025913Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.886044403Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.886093185Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.886252038Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.886434093Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.886572766Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.886821583Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.887188572Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.887457848Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.887501659Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.887567851Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.887616502Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.887716824Z namespaces/openshift-ingress/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.887769696Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.887873268Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.88794405Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.888009952Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.888066193Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.888151145Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.88837355Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.888645837Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.888702569Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.88874532Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.888822481Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.888861662Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.888975735Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.889044997Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.889083638Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.889200551Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.889257952Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.889267363Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.889330984Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.889443907Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.889549999Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.889600831Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.890092853Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.890210246Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.890489182Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.890573405Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.89081183Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.890967744Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.89119957Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.891301142Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.891332143Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.891431626Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.891446696Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.891499447Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.891586529Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.89162203Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.891693472Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.891739153Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.891827055Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.891928328Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.891988179Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.892043001Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.892088502Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.892137223Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.892189194Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.892229745Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.892278917Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.892312287Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.892354488Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.892527603Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.892725497Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.892858991Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.893118337Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.893374033Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.893575849Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.893583219Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.89363716Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/istiod-openshift-gateway-55ff986f96-fjvbp.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.893747173Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/discovery/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.893754733Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/discovery/discovery/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.893765363Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/discovery/discovery/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.893822735Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/discovery/discovery/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.895609028Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.89568769Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-55ff986f96-fjvbp/discovery/discovery/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.895740342Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.895784083Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.895914946Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/istio-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.895927516Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/istio-proxy/istio-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.895936127Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.895985568Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.896177113Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.896253304Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-6b94bb86d8-j64f2/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.896315536Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.896366357Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/router-default-747f569c94-mjbpd.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.89646911Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/router/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.89647755Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/router/router/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.89648835Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/router/router/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.896533101Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/router/router/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.896650824Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/router/router/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.896724696Z namespaces/openshift-ingress/pods/router-default-747f569c94-mjbpd/router/router/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.896765457Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.896809448Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.89688891Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.896983132Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.897100135Z namespaces/openshift-insights/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.897154177Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.897228058Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.897264999Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.897342011Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.897383562Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.897510275Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.897623978Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.897736171Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.897798312Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.897849094Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.897951126Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.897988607Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.898049439Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.89810032Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.898188922Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.898238683Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.898274594Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.898364836Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.898436138Z namespaces/openshift-insights/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.898488139Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.899385591Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.899476134Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.89974516Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.899916754Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.900295924Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.900390006Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.90053816Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.900617592Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.900666723Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.900749085Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.900794956Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.900892578Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.900998411Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.901019972Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.901084093Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.901134284Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.901200886Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.901256697Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.901313919Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.90135409Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.901454082Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.901528294Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.901590266Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.901653837Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.901733189Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.901796621Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.901879983Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.901977485Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.902087818Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.902107058Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.90219048Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.902234471Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.902285973Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.902463187Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.902672382Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.902848027Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.903212415Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.903528923Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.90378814Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.90379552Z namespaces/openshift-insights/pods/insights-operator-585dfdc468-6gf74/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.903845571Z namespaces/openshift-insights/pods/insights-operator-585dfdc468-6gf74/insights-operator-585dfdc468-6gf74.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.903969414Z namespaces/openshift-insights/pods/insights-operator-585dfdc468-6gf74/insights-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.903986054Z namespaces/openshift-insights/pods/insights-operator-585dfdc468-6gf74/insights-operator/insights-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.903990405Z namespaces/openshift-insights/pods/insights-operator-585dfdc468-6gf74/insights-operator/insights-operator/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.904028626Z namespaces/openshift-insights/pods/insights-operator-585dfdc468-6gf74/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.904506027Z namespaces/openshift-insights/pods/insights-operator-585dfdc468-6gf74/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.904865206Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.904942048Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/insights-runtime-extractor-cl2hj.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.905069751Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/exporter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.905081312Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/exporter/exporter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.905090012Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/exporter/exporter/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.905170414Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/exporter/exporter/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.905280256Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.905370999Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/exporter/exporter/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.90542026Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/extractor/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.90542918Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/extractor/extractor/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.905476421Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/extractor/extractor/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.905539483Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/extractor/extractor/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.905667746Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.905765418Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/extractor/extractor/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.905813379Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.90582178Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.90582617Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.905873151Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.906014634Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.906107057Z namespaces/openshift-insights/pods/insights-runtime-extractor-cl2hj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.906443935Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.906513517Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/insights-runtime-extractor-dpxlz.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.906619879Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/exporter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.906626739Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/exporter/exporter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.906629999Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/exporter/exporter/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.906692801Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/exporter/exporter/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.906771823Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.906851635Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/exporter/exporter/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.906929277Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/extractor/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.906951758Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/extractor/extractor/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.906969698Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/extractor/extractor/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.90706858Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/extractor/extractor/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.907232604Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.907354357Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/extractor/extractor/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.907428139Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.907439799Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.90744908Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.907500021Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.907605474Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.907680636Z namespaces/openshift-insights/pods/insights-runtime-extractor-dpxlz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.907740957Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.907794578Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/insights-runtime-extractor-wsvkb.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.90789013Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/exporter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.907918421Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/exporter/exporter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.907928142Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/exporter/exporter/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.907984063Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/exporter/exporter/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908082235Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908157417Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/exporter/exporter/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908197838Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/extractor/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908204168Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/extractor/extractor/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908207538Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/extractor/extractor/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.90825451Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/extractor/extractor/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908353912Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908427694Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/extractor/extractor/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908475805Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908487135Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908499556Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908535186Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908628989Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908711531Z namespaces/openshift-insights/pods/insights-runtime-extractor-wsvkb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908748462Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908792753Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908857954Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.908926176Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909041259Z namespaces/openshift-keda/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909048909Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909059009Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.90911102Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909209223Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909272424Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909278965Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909322886Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909399118Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909443669Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909542901Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909631583Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909702025Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909710835Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909768257Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.909948771Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.910154926Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.18.1-2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.910341721Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.910562926Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.91071404Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.910921325Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.911147991Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.911191822Z namespaces/openshift-keda/operators.coreos.com/installplans/install-88pk8.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.911295284Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.911329075Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.18.1-2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.911396807Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.911432218Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.911481879Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.91153012Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.911628702Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.911681984Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.911741665Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.911787956Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.911863128Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.9119313Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.912027502Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.912116834Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.912199627Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.912246338Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.912303489Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.91235353Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.912393471Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.912490614Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.912542285Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.912598926Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.912685328Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.91276038Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.912805502Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.912967106Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.913057008Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.91314455Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.913233812Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.913313204Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.913447147Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.913618211Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.913687453Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.913729884Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.913794976Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.913829627Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.913921599Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.91396908Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.914061962Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.914148564Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.914208206Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.914254987Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.914319879Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.914329999Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.914403121Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.914445402Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.914485913Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.914646067Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.914828031Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.915004805Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.915220201Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.915462647Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.915675622Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.915723343Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.915792225Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.915824325Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.915919758Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.91599607Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.916064782Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.916118813Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.916186614Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.916226675Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.916318968Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.9164013Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.916490272Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.916543693Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.916595264Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.916650846Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.916695357Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.91679516Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.916835781Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.916879932Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.917088897Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.9172043Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.917265061Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.917399775Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.917506147Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.91761183Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.917714192Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.917818815Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.917984859Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.918152433Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.918270536Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.918312707Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.918364188Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.918410909Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.918479561Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.918522522Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.918613714Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.918696496Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.918755768Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.918812299Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.918886181Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.918952673Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.919067625Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.919160268Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.919205729Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.91926687Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.919314991Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.919373503Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.919386493Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.919425904Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.919581888Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.919768413Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.919924666Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.920172052Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.920413638Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.920610913Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.920643594Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.920715746Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.920743046Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.920844439Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.92089116Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.921009373Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.921059104Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.921130136Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.921194378Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.92128926Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.921371892Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.921469724Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.921604488Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.921649419Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.92169905Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.921746281Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.921836953Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.921890015Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.921963987Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.922058729Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.92211802Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.922166532Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.922280144Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.922365937Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.922452369Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.922536131Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.922622423Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.922754046Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.922973001Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923038923Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923090164Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923144686Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923193327Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923248458Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.92333643Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923436573Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923521485Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923571386Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923611097Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923671869Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.92371842Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923775431Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923823952Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923872494Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923885444Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.923956726Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.924117529Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.924302164Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.924445088Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.924651233Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.924891389Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.925135025Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.925155215Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.925219777Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.925266868Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.92533993Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.925385211Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.925443152Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.925482073Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.925537304Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.925583556Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.925668918Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.92575484Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.925841172Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.925888313Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.925961115Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.926030796Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.926064797Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.92615372Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.926200371Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.926248342Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.926337844Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.926382905Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.926389645Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.926441077Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.926499518Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.926547099Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.926639662Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.926717644Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.927027721Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.927114273Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.927197505Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.927327459Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.927494093Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.927548754Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.927595285Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.927646307Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.927690017Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.927739419Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.92778538Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.927872042Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.927987045Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.928044116Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.928085397Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.928133378Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.928179289Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.928230671Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.928239001Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.928287982Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.928438326Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.92861962Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.928762744Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.92900509Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.929249876Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.929443551Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.929481432Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.929537013Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.929573784Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.929636655Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.929681217Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.929746758Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.929785819Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.929852751Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.929895972Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.930015185Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.930097407Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.930186119Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.93023283Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.930286801Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.930329933Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.930382874Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.930464836Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.930525967Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.930556328Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.93065415Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.930714992Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.930763513Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.930878976Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.930990869Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.931073531Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.931162943Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.931252155Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.931370408Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.931540112Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.931593133Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.931641875Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.931687126Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.931733747Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.931782198Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.931819299Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.931935982Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.932030934Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.932086985Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.932135867Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.932189148Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.932240989Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.932296491Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.932312061Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.932367532Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.932525706Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.932718391Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.932858885Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.933124021Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.933433379Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.933707015Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.933753126Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.933822298Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.9338725Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.933995463Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934050064Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934124236Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934169757Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934219618Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934265179Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934359901Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934445854Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934533016Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934581897Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934633758Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934682069Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934726981Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934810352Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934867814Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.934919355Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.935021418Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.935076629Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.93512496Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.935227543Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.935314035Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.935433638Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.93552421Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.935636523Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.935772346Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.935959291Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936030623Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936078594Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936117235Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936195867Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936250228Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936301729Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936393202Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936476084Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936522755Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936569026Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936649518Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936694299Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936765461Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936773971Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.936831292Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.937034217Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.937230592Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.937385566Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.937588461Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.937832907Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.938050472Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.938072673Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.938138654Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.938166015Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.938271228Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.938306729Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.93838Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.938431052Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.938501993Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.938548144Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.938635797Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.938747339Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.938863582Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.938926994Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.938949214Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.939010596Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.939059607Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.939151889Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.9392024Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.939245812Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.939332024Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.939381795Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.939393765Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.939433026Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.939481998Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.939526139Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.939632171Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.939725844Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.939885557Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.93999709Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.940123423Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.940245456Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.94041789Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.940488282Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.940533213Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.940594195Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.940635516Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.940729608Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.940764709Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.940852541Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.941029805Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.941087177Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.941133448Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.941180809Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.941227181Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.941279932Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.941286912Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.941327063Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.941487777Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.941673471Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.941815755Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.942045621Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.942292767Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.942490121Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.942497422Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6769c5d45-6nd9l/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.942536653Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6769c5d45-6nd9l/kube-storage-version-migrator-operator-6769c5d45-6nd9l.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.942661286Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6769c5d45-6nd9l/kube-storage-version-migrator-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.942669486Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6769c5d45-6nd9l/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.942673036Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6769c5d45-6nd9l/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.942717207Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6769c5d45-6nd9l/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.94284577Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6769c5d45-6nd9l/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943011744Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943029745Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943105667Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943135257Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943217579Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.94326009Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943320142Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943365503Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943418554Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943459705Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943547437Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.94365683Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943764863Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943796893Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943843585Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943888156Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.943958717Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.94405232Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.944111301Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.944146972Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.944238014Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.944294386Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.944347297Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.94445252Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.944530722Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.944651045Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.944732776Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.94485471Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945013994Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945180558Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945235069Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.94528507Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945338531Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945394653Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945429114Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945478535Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945568387Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945647529Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.94569289Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945738771Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945773642Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945822534Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945873675Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945881465Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.945955847Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.94610282Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.946305235Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.946444529Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.946657264Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.94691359Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947114515Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947124195Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-74bb7799d9-d6bzf/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947157256Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-74bb7799d9-d6bzf/migrator-74bb7799d9-d6bzf.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947258429Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-74bb7799d9-d6bzf/graceful-termination/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947264949Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-74bb7799d9-d6bzf/graceful-termination/graceful-termination/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947273789Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-74bb7799d9-d6bzf/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.94732364Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-74bb7799d9-d6bzf/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947417143Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-74bb7799d9-d6bzf/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947486034Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-74bb7799d9-d6bzf/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947529115Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-74bb7799d9-d6bzf/migrator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947535945Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-74bb7799d9-d6bzf/migrator/migrator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947539276Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-74bb7799d9-d6bzf/migrator/migrator/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947584037Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-74bb7799d9-d6bzf/migrator/migrator/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947688329Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-74bb7799d9-d6bzf/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947761951Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-74bb7799d9-d6bzf/migrator/migrator/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947805852Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947849023Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947917915Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.947976156Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.948039108Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.948093329Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.948146471Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.948194222Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.948256993Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.948301834Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.948396167Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.94853415Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.948662013Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.948711524Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.948762546Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.948807537Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.948856748Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.949055203Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.949109924Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.949159195Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.949236357Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.949286428Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.949293099Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.94936172Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.949470243Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.949538855Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.949574726Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.949688429Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.949780241Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.950116559Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.950196281Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.950382726Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.950512659Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.950704063Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.950770625Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.950820376Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.950887988Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.9509514Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.951008781Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.951052452Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.951137344Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.951230547Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.951275508Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.951329269Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.95138103Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.951429211Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.951491983Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.951528764Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.951583245Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.951594725Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.951635646Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.951806941Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.952013976Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.952161379Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.952447516Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.952623521Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.952838646Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953073142Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953122513Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-twczm.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953200895Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953243906Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953300677Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953347368Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953387949Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953440771Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953511392Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953518903Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-77s29/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953562244Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-77s29/lws-controller-manager-7bc7774745-77s29.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953654986Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-77s29/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953662686Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-77s29/manager/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953666766Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-77s29/manager/manager/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953705147Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-77s29/manager/manager/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.95382079Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-77s29/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953897672Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-77s29/manager/manager/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.953964654Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-kjk2m/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954018425Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-kjk2m/lws-controller-manager-7bc7774745-kjk2m.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954104907Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-kjk2m/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954115417Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-kjk2m/manager/manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954127818Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-kjk2m/manager/manager/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954160268Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-kjk2m/manager/manager/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954323923Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-kjk2m/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954399594Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7bc7774745-kjk2m/manager/manager/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954448075Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-bfc7f696d-wxdcx/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954498727Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-bfc7f696d-wxdcx/openshift-lws-operator-bfc7f696d-wxdcx.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954592599Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-bfc7f696d-wxdcx/openshift-lws-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954601149Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-bfc7f696d-wxdcx/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954604599Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-bfc7f696d-wxdcx/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.95464524Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-bfc7f696d-wxdcx/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954826565Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-bfc7f696d-wxdcx/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954913377Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-bfc7f696d-wxdcx/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.954962638Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.955017909Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.955068431Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.955117602Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.955169373Z namespaces/openshift-machine-api/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.955175633Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.955181253Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.955221634Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.955384729Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.955570133Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.955719517Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.955955763Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.956216539Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.956424314Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.956433934Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.956439864Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.956481636Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.956586708Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.95665602Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.95667916Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.956698291Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.956887106Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.957183813Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.957317556Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.957550402Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.957797558Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.958006193Z namespaces/openshift-marketplace/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.958018383Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.958022003Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.958067075Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.958139446Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.958148317Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.958197018Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.9582929Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.958384162Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.958472405Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.958538316Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.958570247Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.958733861Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.958934776Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.95908935Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.959279714Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.95952461Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.959737525Z namespaces/openshift-monitoring/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.959786517Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.959863619Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.95991478Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.959971271Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.960026503Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.960162496Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.960597287Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.960980606Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.961205322Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.961252723Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.961305774Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.961346985Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.961429517Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.961480419Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.961526249Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.961613472Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.961671893Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.961725465Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.964714358Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.964859231Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.965981769Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.966064571Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.966994284Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.967123217Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.967589009Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.967771933Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.967815864Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.967970328Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.967994089Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.968065821Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.968104641Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.968196764Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.968274835Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.968333767Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.968383688Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.968531012Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.968568923Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.968664185Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.968712286Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.968811029Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.96884791Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.968992833Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.969154977Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.969246519Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.969563207Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.969699201Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.969828034Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.969940556Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.970046559Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.970146761Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.970218093Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.970264404Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.970360317Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.970449489Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.970538341Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.970643924Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.970732546Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.97088565Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.971044424Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.971143636Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.971236748Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.971332291Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.971425683Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.971479954Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.971527575Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.971572787Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.971584627Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.971638998Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.971835693Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.972084899Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.972236793Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.972442178Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.972704304Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973061013Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973085914Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973198807Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973207647Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973273029Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973433733Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973445493Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973451433Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973482484Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973585046Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973660618Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973693919Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973701159Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973705429Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.97374642Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973853483Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973946765Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973989556Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.973998086Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974002216Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974051718Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.97414906Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974219892Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974260903Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974271543Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974275983Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974310474Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974410477Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974486508Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974529959Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.97454107Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.97454798Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974587101Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974680033Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974754285Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974795976Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974803776Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974810946Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.974841217Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.97495431Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.975029992Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.975073033Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.975079423Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.975082803Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.975128724Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.975219666Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.975293888Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.975339129Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-75587bd455-5cln8/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.97538391Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-75587bd455-5cln8/cluster-monitoring-operator-75587bd455-5cln8.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.975473953Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-75587bd455-5cln8/cluster-monitoring-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.975480823Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-75587bd455-5cln8/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.975484633Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-75587bd455-5cln8/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.975525884Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-75587bd455-5cln8/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.976261032Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-75587bd455-5cln8/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.976341334Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-75587bd455-5cln8/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.976381135Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.976429596Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-state-metrics-69db897b98-7vn6j.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.976526989Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-rbac-proxy-main/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.976537199Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.976542259Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.9765843Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.976684642Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.976764764Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.976808965Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-rbac-proxy-self/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.976816105Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.976819616Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.976863577Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.97699741Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977077322Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977113943Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-state-metrics/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977126163Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977137353Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977184125Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977349749Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977429241Z namespaces/openshift-monitoring/pods/kube-state-metrics-69db897b98-7vn6j/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977480492Z namespaces/openshift-monitoring/pods/metrics-server-56c8457455-7bjzd/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977522003Z namespaces/openshift-monitoring/pods/metrics-server-56c8457455-7bjzd/metrics-server-56c8457455-7bjzd.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977620595Z namespaces/openshift-monitoring/pods/metrics-server-56c8457455-7bjzd/metrics-server/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977629925Z namespaces/openshift-monitoring/pods/metrics-server-56c8457455-7bjzd/metrics-server/metrics-server/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977634886Z namespaces/openshift-monitoring/pods/metrics-server-56c8457455-7bjzd/metrics-server/metrics-server/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977679067Z namespaces/openshift-monitoring/pods/metrics-server-56c8457455-7bjzd/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977782999Z namespaces/openshift-monitoring/pods/metrics-server-56c8457455-7bjzd/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977864051Z namespaces/openshift-monitoring/pods/metrics-server-56c8457455-7bjzd/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977927983Z namespaces/openshift-monitoring/pods/monitoring-plugin-7dccd58f55-j7zhr/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.977971744Z namespaces/openshift-monitoring/pods/monitoring-plugin-7dccd58f55-j7zhr/monitoring-plugin-7dccd58f55-j7zhr.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978060336Z namespaces/openshift-monitoring/pods/monitoring-plugin-7dccd58f55-j7zhr/monitoring-plugin/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978069807Z namespaces/openshift-monitoring/pods/monitoring-plugin-7dccd58f55-j7zhr/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978074856Z namespaces/openshift-monitoring/pods/monitoring-plugin-7dccd58f55-j7zhr/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978113337Z namespaces/openshift-monitoring/pods/monitoring-plugin-7dccd58f55-j7zhr/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.97821403Z namespaces/openshift-monitoring/pods/monitoring-plugin-7dccd58f55-j7zhr/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978289972Z namespaces/openshift-monitoring/pods/monitoring-plugin-7dccd58f55-j7zhr/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978319113Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978367914Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/node-exporter-4jj65.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978473886Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/init-textfile/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978483107Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/init-textfile/init-textfile/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978489847Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/init-textfile/init-textfile/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978540478Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.97861916Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978697792Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978752443Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978763973Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978767754Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978802414Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978912497Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.978999389Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.97903915Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/node-exporter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.97904642Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/node-exporter/node-exporter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979049691Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/node-exporter/node-exporter/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979092782Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979205604Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979279096Z namespaces/openshift-monitoring/pods/node-exporter-4jj65/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979326637Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979368678Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/node-exporter-d67bp.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979473781Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/init-textfile/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979480911Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/init-textfile/init-textfile/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979484391Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/init-textfile/init-textfile/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979531802Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979610804Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979688546Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979730857Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979744157Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979752488Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979802369Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979914742Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.979997144Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980036655Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/node-exporter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980043075Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/node-exporter/node-exporter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980056995Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/node-exporter/node-exporter/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980090106Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980202089Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980276691Z namespaces/openshift-monitoring/pods/node-exporter-d67bp/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980324142Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980376513Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/node-exporter-swxxb.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980464235Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/init-textfile/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980473775Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/init-textfile/init-textfile/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980477246Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/init-textfile/init-textfile/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980520947Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980603729Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.98067594Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980716852Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980723062Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980726602Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980768073Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980856845Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.980995559Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.9810372Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/node-exporter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.98104422Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/node-exporter/node-exporter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.98104762Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/node-exporter/node-exporter/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981086941Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981205724Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981280366Z namespaces/openshift-monitoring/pods/node-exporter-swxxb/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981331387Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981375718Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/openshift-state-metrics-9d44df66c-cms96.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981504301Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/kube-rbac-proxy-main/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981510861Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981514231Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981563522Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981660335Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981738867Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981776468Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/kube-rbac-proxy-self/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981782938Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981789958Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981829789Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.981945842Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982024894Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982064875Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/openshift-state-metrics/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982071955Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982080505Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982125126Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982230959Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982307931Z namespaces/openshift-monitoring/pods/openshift-state-metrics-9d44df66c-cms96/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982374432Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982444904Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982611888Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982621318Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982635169Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.98268582Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982784632Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982860874Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982916986Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982928246Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982939366Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.982991377Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.98308123Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983156992Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983206603Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983212953Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983222413Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983265874Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983364507Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983436248Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.98348678Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.98349325Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.98350197Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983545831Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983635323Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983718595Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983755556Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983762796Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983768137Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983812468Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983925631Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.983995892Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.984037523Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.984047714Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.984057814Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.984100145Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.984873894Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.984986106Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985029998Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985037048Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985040768Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.98511389Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985222742Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985303584Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985333705Z namespaces/openshift-monitoring/pods/prometheus-operator-5676c8c784-9dbkc/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985384176Z namespaces/openshift-monitoring/pods/prometheus-operator-5676c8c784-9dbkc/prometheus-operator-5676c8c784-9dbkc.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985474538Z namespaces/openshift-monitoring/pods/prometheus-operator-5676c8c784-9dbkc/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985481159Z namespaces/openshift-monitoring/pods/prometheus-operator-5676c8c784-9dbkc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985484539Z namespaces/openshift-monitoring/pods/prometheus-operator-5676c8c784-9dbkc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.98552794Z namespaces/openshift-monitoring/pods/prometheus-operator-5676c8c784-9dbkc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985627222Z namespaces/openshift-monitoring/pods/prometheus-operator-5676c8c784-9dbkc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985701204Z namespaces/openshift-monitoring/pods/prometheus-operator-5676c8c784-9dbkc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985732735Z namespaces/openshift-monitoring/pods/prometheus-operator-5676c8c784-9dbkc/prometheus-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985739475Z namespaces/openshift-monitoring/pods/prometheus-operator-5676c8c784-9dbkc/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985746805Z namespaces/openshift-monitoring/pods/prometheus-operator-5676c8c784-9dbkc/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.985799326Z namespaces/openshift-monitoring/pods/prometheus-operator-5676c8c784-9dbkc/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986077803Z namespaces/openshift-monitoring/pods/prometheus-operator-5676c8c784-9dbkc/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986156555Z namespaces/openshift-monitoring/pods/prometheus-operator-5676c8c784-9dbkc/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986199816Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-57cf98b594-2jncb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986249628Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-57cf98b594-2jncb/prometheus-operator-admission-webhook-57cf98b594-2jncb.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986368711Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-57cf98b594-2jncb/prometheus-operator-admission-webhook/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986375721Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-57cf98b594-2jncb/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986379231Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-57cf98b594-2jncb/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986426622Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-57cf98b594-2jncb/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986524175Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-57cf98b594-2jncb/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986604186Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-57cf98b594-2jncb/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986655988Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986710839Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/telemeter-client-7df7f7d9c5-97595.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986854632Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986861763Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986865033Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.986957135Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987054897Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987132069Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.98717207Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/reload/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987178891Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/reload/reload/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987182111Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/reload/reload/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987235232Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/reload/reload/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987335324Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987409976Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/reload/reload/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987458777Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/telemeter-client/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987470898Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/telemeter-client/telemeter-client/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987476468Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987520729Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987725014Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987807616Z namespaces/openshift-monitoring/pods/telemeter-client-7df7f7d9c5-97595/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987851707Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.987954059Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/thanos-querier-5cd7c4cf98-6qxq2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988103933Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-metrics/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988110923Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988114254Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988166265Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988264757Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988337099Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.98838363Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-rules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.98839108Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.98840103Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988451882Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988549674Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988623616Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988668157Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-web/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988674897Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988678227Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988715948Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988815891Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.988886513Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989016936Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989029176Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989032676Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989061797Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989158819Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989233661Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989301753Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/prom-label-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989307923Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989311333Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989355094Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989449966Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989526298Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989566439Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/thanos-query/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989572859Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/thanos-query/thanos-query/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989581019Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/thanos-query/thanos-query/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989643841Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989744554Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989817555Z namespaces/openshift-monitoring/pods/thanos-querier-5cd7c4cf98-6qxq2/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989861296Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989919768Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.989976869Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.99002113Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.990127793Z namespaces/openshift-multus/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.990173344Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.990242226Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.990283617Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.990342118Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.990383769Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.990625515Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.990712257Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.99079754Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.99084127Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.990895142Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.991054326Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.991173399Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.99124054Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.991285241Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.991342783Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.991383214Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.991471906Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.991519577Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.991564868Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.991653021Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.991721332Z namespaces/openshift-multus/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.991764083Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.991932717Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.99202832Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.992513172Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.992651835Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.993293211Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.993410814Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.993567357Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.993628589Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.99367809Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.993776483Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.993863435Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.993984228Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.994128841Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.994164702Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.994270145Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.994356137Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.994448909Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.9944962Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.994544701Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.994601083Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.994639994Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.994698305Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.994749066Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.994797498Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.994843719Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.994936061Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.995035264Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.995081315Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.995127556Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.995185447Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.995233218Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.99529739Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.995348401Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.995421683Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.995433123Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.995485305Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.995668049Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.995889705Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.996066589Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.996279154Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.99651977Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.996739676Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.996752016Z namespaces/openshift-multus/pods/multus-9c58k/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.996785127Z namespaces/openshift-multus/pods/multus-9c58k/multus-9c58k.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.996974232Z namespaces/openshift-multus/pods/multus-9c58k/kube-multus/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.996989022Z namespaces/openshift-multus/pods/multus-9c58k/kube-multus/kube-multus/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.997050693Z namespaces/openshift-multus/pods/multus-9c58k/kube-multus/kube-multus/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.997113185Z namespaces/openshift-multus/pods/multus-9c58k/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.997565266Z namespaces/openshift-multus/pods/multus-9c58k/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.997644178Z namespaces/openshift-multus/pods/multus-9c58k/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.997697039Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.997754611Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/multus-additional-cni-plugins-5s6bg.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.997888934Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/bond-cni-plugin/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.997895994Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.997915484Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.997973956Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998066848Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.99814165Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998183551Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/cni-plugins/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998189961Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/cni-plugins/cni-plugins/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998193351Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998233492Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998339235Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998408097Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998448408Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/egress-router-binary-copy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998454708Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998462438Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998506729Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998603892Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998682063Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998725035Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998732775Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998738135Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998780366Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.998854978Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.99896625Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999015662Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/routeoverride-cni/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999023212Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999028822Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999068043Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999161105Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999234467Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999271428Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/whereabouts-cni-bincopy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999279018Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999282358Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999325989Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999417472Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999493133Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999535784Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/whereabouts-cni/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999545484Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999589626Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999642057Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999736639Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999811891Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-5s6bg/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999855632Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:33.999918994Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/multus-additional-cni-plugins-d7svv.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000073308Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/bond-cni-plugin/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000086838Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000090488Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000125689Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000217031Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000288463Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000334354Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/cni-plugins/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000342594Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/cni-plugins/cni-plugins/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000349874Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000378975Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000471687Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000547019Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00058851Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/egress-router-binary-copy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.0005947Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00060011Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000634551Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000730544Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000807556Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000847367Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000854367Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000857687Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.000896638Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001020691Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001098193Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001143934Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/routeoverride-cni/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001152894Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001158584Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001188395Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001282637Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001362539Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00140043Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/whereabouts-cni-bincopy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00140655Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00140977Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001452871Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001542884Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001616486Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001656487Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/whereabouts-cni/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001662837Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001666267Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001705718Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00179712Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.001872092Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-d7svv/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002035006Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002088637Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/multus-additional-cni-plugins-xcnjl.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002231541Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/bond-cni-plugin/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002239671Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002243111Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002280942Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002376094Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002451776Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002495267Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/cni-plugins/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002503417Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/cni-plugins/cni-plugins/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002509117Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002556119Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002641781Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002722933Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002762724Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/egress-router-binary-copy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002769954Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002775784Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002820355Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.002933198Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00302882Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003080102Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003092112Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003096122Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003122983Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003199985Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003284367Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003320497Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/routeoverride-cni/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003330588Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003339498Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003392819Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003482641Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003556283Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003597034Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/whereabouts-cni-bincopy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003604174Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003607664Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003642555Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003739058Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00381318Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003853961Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/whereabouts-cni/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003860111Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003863361Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.003919602Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.004029695Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.004102387Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xcnjl/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.004145158Z namespaces/openshift-multus/pods/multus-kl27l/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.004183739Z namespaces/openshift-multus/pods/multus-kl27l/multus-kl27l.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.004285231Z namespaces/openshift-multus/pods/multus-kl27l/kube-multus/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.004294451Z namespaces/openshift-multus/pods/multus-kl27l/kube-multus/kube-multus/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.004298152Z namespaces/openshift-multus/pods/multus-kl27l/kube-multus/kube-multus/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.004325242Z namespaces/openshift-multus/pods/multus-kl27l/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.004885086Z namespaces/openshift-multus/pods/multus-kl27l/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.004999169Z namespaces/openshift-multus/pods/multus-kl27l/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.005033859Z namespaces/openshift-multus/pods/multus-lvm6f/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.005081891Z namespaces/openshift-multus/pods/multus-lvm6f/multus-lvm6f.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.005170763Z namespaces/openshift-multus/pods/multus-lvm6f/kube-multus/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.005178293Z namespaces/openshift-multus/pods/multus-lvm6f/kube-multus/kube-multus/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.005182303Z namespaces/openshift-multus/pods/multus-lvm6f/kube-multus/kube-multus/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.005216784Z namespaces/openshift-multus/pods/multus-lvm6f/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.005724387Z namespaces/openshift-multus/pods/multus-lvm6f/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.005804749Z namespaces/openshift-multus/pods/multus-lvm6f/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00584511Z namespaces/openshift-multus/pods/network-metrics-daemon-76492/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00588794Z namespaces/openshift-multus/pods/network-metrics-daemon-76492/network-metrics-daemon-76492.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.005999573Z namespaces/openshift-multus/pods/network-metrics-daemon-76492/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006013014Z namespaces/openshift-multus/pods/network-metrics-daemon-76492/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006016694Z namespaces/openshift-multus/pods/network-metrics-daemon-76492/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006053015Z namespaces/openshift-multus/pods/network-metrics-daemon-76492/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006142517Z namespaces/openshift-multus/pods/network-metrics-daemon-76492/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006228469Z namespaces/openshift-multus/pods/network-metrics-daemon-76492/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00626341Z namespaces/openshift-multus/pods/network-metrics-daemon-76492/network-metrics-daemon/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00627127Z namespaces/openshift-multus/pods/network-metrics-daemon-76492/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00627538Z namespaces/openshift-multus/pods/network-metrics-daemon-76492/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006317151Z namespaces/openshift-multus/pods/network-metrics-daemon-76492/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006463295Z namespaces/openshift-multus/pods/network-metrics-daemon-76492/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006539817Z namespaces/openshift-multus/pods/network-metrics-daemon-76492/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006581288Z namespaces/openshift-multus/pods/network-metrics-daemon-dtrg9/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006622829Z namespaces/openshift-multus/pods/network-metrics-daemon-dtrg9/network-metrics-daemon-dtrg9.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00669964Z namespaces/openshift-multus/pods/network-metrics-daemon-dtrg9/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006705951Z namespaces/openshift-multus/pods/network-metrics-daemon-dtrg9/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006709281Z namespaces/openshift-multus/pods/network-metrics-daemon-dtrg9/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006749112Z namespaces/openshift-multus/pods/network-metrics-daemon-dtrg9/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006844374Z namespaces/openshift-multus/pods/network-metrics-daemon-dtrg9/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006948927Z namespaces/openshift-multus/pods/network-metrics-daemon-dtrg9/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.006992298Z namespaces/openshift-multus/pods/network-metrics-daemon-dtrg9/network-metrics-daemon/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007000188Z namespaces/openshift-multus/pods/network-metrics-daemon-dtrg9/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007012448Z namespaces/openshift-multus/pods/network-metrics-daemon-dtrg9/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007053259Z namespaces/openshift-multus/pods/network-metrics-daemon-dtrg9/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007191113Z namespaces/openshift-multus/pods/network-metrics-daemon-dtrg9/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007274425Z namespaces/openshift-multus/pods/network-metrics-daemon-dtrg9/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007315676Z namespaces/openshift-multus/pods/network-metrics-daemon-vjjmx/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007364147Z namespaces/openshift-multus/pods/network-metrics-daemon-vjjmx/network-metrics-daemon-vjjmx.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007440659Z namespaces/openshift-multus/pods/network-metrics-daemon-vjjmx/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007447249Z namespaces/openshift-multus/pods/network-metrics-daemon-vjjmx/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007453939Z namespaces/openshift-multus/pods/network-metrics-daemon-vjjmx/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00750526Z namespaces/openshift-multus/pods/network-metrics-daemon-vjjmx/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007606123Z namespaces/openshift-multus/pods/network-metrics-daemon-vjjmx/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007679735Z namespaces/openshift-multus/pods/network-metrics-daemon-vjjmx/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007728886Z namespaces/openshift-multus/pods/network-metrics-daemon-vjjmx/network-metrics-daemon/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007737386Z namespaces/openshift-multus/pods/network-metrics-daemon-vjjmx/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007740696Z namespaces/openshift-multus/pods/network-metrics-daemon-vjjmx/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007773867Z namespaces/openshift-multus/pods/network-metrics-daemon-vjjmx/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.007979562Z namespaces/openshift-multus/pods/network-metrics-daemon-vjjmx/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.008059294Z namespaces/openshift-multus/pods/network-metrics-daemon-vjjmx/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.008101755Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.008142776Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.008196557Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.008204277Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.008246269Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.008340511Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.008392252Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.008468394Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.008559866Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.008617448Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.008669169Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.0087169Z namespaces/openshift-must-gather-j4sk6/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00872301Z namespaces/openshift-must-gather-j4sk6/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.00872632Z namespaces/openshift-must-gather-j4sk6/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.008768971Z namespaces/openshift-must-gather-j4sk6/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.009028928Z namespaces/openshift-must-gather-j4sk6/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.009246483Z namespaces/openshift-must-gather-j4sk6/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.009399197Z namespaces/openshift-must-gather-j4sk6/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.009606472Z namespaces/openshift-must-gather-j4sk6/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.009843228Z namespaces/openshift-must-gather-j4sk6/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.010075554Z namespaces/openshift-network-console/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.010129035Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.010198317Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.010235698Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.010290929Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01033422Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.010422762Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.010557755Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.010671488Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.010713379Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.010762331Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.010833332Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.010876603Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.010959125Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.011026557Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.011109549Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01116639Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.011207631Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.011290873Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.011336565Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.011381916Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.011510089Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.011592011Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.011720144Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.011804826Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01197514Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.012110584Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.012251217Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.012314499Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01236636Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.012423701Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.012472293Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.012526944Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.012570965Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.012624736Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.012669457Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.012716519Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01276659Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.012848892Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.012973525Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.013021486Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.013071987Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.013120238Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01316793Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.013229971Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.013236321Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.013277732Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.013433406Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.013618731Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.013757114Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01398863Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.014241196Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.014439361Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.014448421Z namespaces/openshift-network-console/pods/networking-console-plugin-cb95c66f6-mndcn/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.014481872Z namespaces/openshift-network-console/pods/networking-console-plugin-cb95c66f6-mndcn/networking-console-plugin-cb95c66f6-mndcn.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.014566384Z namespaces/openshift-network-console/pods/networking-console-plugin-cb95c66f6-mndcn/networking-console-plugin/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.014575104Z namespaces/openshift-network-console/pods/networking-console-plugin-cb95c66f6-mndcn/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.014578644Z namespaces/openshift-network-console/pods/networking-console-plugin-cb95c66f6-mndcn/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.014612395Z namespaces/openshift-network-console/pods/networking-console-plugin-cb95c66f6-mndcn/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.014710438Z namespaces/openshift-network-console/pods/networking-console-plugin-cb95c66f6-mndcn/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01478465Z namespaces/openshift-network-console/pods/networking-console-plugin-cb95c66f6-mndcn/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.014838051Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.014869901Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.014952743Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.014985314Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015046146Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015095657Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015146708Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015178969Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015248391Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015279662Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015393304Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015498697Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01560221Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015643631Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015687561Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015756663Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015807395Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015874096Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015935878Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.015991619Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01604302Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.016128312Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.016175013Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.016221625Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.016306037Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.016356228Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.016401899Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.016506192Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.016598844Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.016794859Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.016878151Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.017173898Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.017267731Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.017373043Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.017435815Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.017471485Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.017530827Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.017578048Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01766964Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.017717982Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.017766163Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.017831674Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.017875915Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.017942747Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.017985078Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01806761Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.018158112Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.018200903Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.018250475Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.018300516Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.018345637Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.018396418Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.018437949Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01848903Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01849523Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.018533662Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.018688005Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01886984Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.019041414Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.019248819Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.019487255Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01968044Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.01968695Z namespaces/openshift-network-diagnostics/pods/network-check-source-8894fc9bd-rltbp/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.019727081Z namespaces/openshift-network-diagnostics/pods/network-check-source-8894fc9bd-rltbp/network-check-source-8894fc9bd-rltbp.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.019814173Z namespaces/openshift-network-diagnostics/pods/network-check-source-8894fc9bd-rltbp/check-endpoints/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.019828223Z namespaces/openshift-network-diagnostics/pods/network-check-source-8894fc9bd-rltbp/check-endpoints/check-endpoints/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.019834704Z namespaces/openshift-network-diagnostics/pods/network-check-source-8894fc9bd-rltbp/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.019859294Z namespaces/openshift-network-diagnostics/pods/network-check-source-8894fc9bd-rltbp/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.019992658Z namespaces/openshift-network-diagnostics/pods/network-check-source-8894fc9bd-rltbp/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020067959Z namespaces/openshift-network-diagnostics/pods/network-check-source-8894fc9bd-rltbp/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.02009317Z namespaces/openshift-network-diagnostics/pods/network-check-target-28blj/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020148041Z namespaces/openshift-network-diagnostics/pods/network-check-target-28blj/network-check-target-28blj.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020226903Z namespaces/openshift-network-diagnostics/pods/network-check-target-28blj/network-check-target-container/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020235233Z namespaces/openshift-network-diagnostics/pods/network-check-target-28blj/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020239473Z namespaces/openshift-network-diagnostics/pods/network-check-target-28blj/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020262974Z namespaces/openshift-network-diagnostics/pods/network-check-target-28blj/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020365757Z namespaces/openshift-network-diagnostics/pods/network-check-target-28blj/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020439629Z namespaces/openshift-network-diagnostics/pods/network-check-target-28blj/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020475929Z namespaces/openshift-network-diagnostics/pods/network-check-target-9p89h/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.02051814Z namespaces/openshift-network-diagnostics/pods/network-check-target-9p89h/network-check-target-9p89h.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020590442Z namespaces/openshift-network-diagnostics/pods/network-check-target-9p89h/network-check-target-container/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020597992Z namespaces/openshift-network-diagnostics/pods/network-check-target-9p89h/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020608842Z namespaces/openshift-network-diagnostics/pods/network-check-target-9p89h/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020642354Z namespaces/openshift-network-diagnostics/pods/network-check-target-9p89h/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020732996Z namespaces/openshift-network-diagnostics/pods/network-check-target-9p89h/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020811367Z namespaces/openshift-network-diagnostics/pods/network-check-target-9p89h/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020837868Z namespaces/openshift-network-diagnostics/pods/network-check-target-xtb9l/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020883269Z namespaces/openshift-network-diagnostics/pods/network-check-target-xtb9l/network-check-target-xtb9l.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.020995492Z namespaces/openshift-network-diagnostics/pods/network-check-target-xtb9l/network-check-target-container/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021008062Z namespaces/openshift-network-diagnostics/pods/network-check-target-xtb9l/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021012352Z namespaces/openshift-network-diagnostics/pods/network-check-target-xtb9l/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021029423Z namespaces/openshift-network-diagnostics/pods/network-check-target-xtb9l/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021141626Z namespaces/openshift-network-diagnostics/pods/network-check-target-xtb9l/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021224208Z namespaces/openshift-network-diagnostics/pods/network-check-target-xtb9l/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021257809Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.02131363Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021355991Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021367811Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021403652Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021487844Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021535865Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021590597Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021669079Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.02172125Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021768881Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021814712Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021864724Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021944266Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.021989076Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022042408Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022084299Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022179531Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022253013Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022340125Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022383356Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022435827Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022481209Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.02252838Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022609522Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022653323Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022703384Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022784666Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022835757Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022845757Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022879038Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.02295796Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.022997161Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.023116664Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.023198536Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.023283858Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.023375921Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.023455432Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.023606236Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.023709109Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.02376352Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.023812091Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.023864353Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.023915154Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.023984686Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.024027257Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.024074028Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.024125549Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.024199191Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.024284343Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.024335164Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.024381395Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.024427816Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.024480378Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.024523199Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.024533229Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.02457588Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.024778075Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.025065892Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.025215536Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.025412791Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.025653257Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.025844861Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.025885862Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.025970305Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.025983935Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026014846Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026056976Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026111828Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026157709Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.02619876Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026248781Z namespaces/openshift-network-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026294462Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026364574Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026406215Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026451846Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026497978Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.02660037Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026680942Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026774424Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026815725Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026866727Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.026935778Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.02699238Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.02702471Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.027075402Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.027164524Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.027212265Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.027249046Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.027333218Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.027391199Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.0273983Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.027440101Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.027490612Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.027541273Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.027659556Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.027746518Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.027919362Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.028007055Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.028186769Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.028275921Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.028407464Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.028463506Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.028511357Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.028600909Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.028661391Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.028707772Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.028760533Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.028804404Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.028864186Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.028923097Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.028982028Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.02902879Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.029138422Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.029223685Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.029271816Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.029303696Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.029387469Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.02943188Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.029499411Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.029516662Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.029591643Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.029610804Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.029673405Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.029681936Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.029720307Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.029879241Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.030121756Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.03027994Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.030490616Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.030751882Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.030997678Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031015309Z namespaces/openshift-network-operator/pods/iptables-alerter-2fswp/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031058519Z namespaces/openshift-network-operator/pods/iptables-alerter-2fswp/iptables-alerter-2fswp.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031161452Z namespaces/openshift-network-operator/pods/iptables-alerter-2fswp/iptables-alerter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031173002Z namespaces/openshift-network-operator/pods/iptables-alerter-2fswp/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031177993Z namespaces/openshift-network-operator/pods/iptables-alerter-2fswp/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031191323Z namespaces/openshift-network-operator/pods/iptables-alerter-2fswp/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031302556Z namespaces/openshift-network-operator/pods/iptables-alerter-2fswp/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031373027Z namespaces/openshift-network-operator/pods/iptables-alerter-2fswp/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031415308Z namespaces/openshift-network-operator/pods/iptables-alerter-9psv4/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031444409Z namespaces/openshift-network-operator/pods/iptables-alerter-9psv4/iptables-alerter-9psv4.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031530191Z namespaces/openshift-network-operator/pods/iptables-alerter-9psv4/iptables-alerter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031539261Z namespaces/openshift-network-operator/pods/iptables-alerter-9psv4/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031544122Z namespaces/openshift-network-operator/pods/iptables-alerter-9psv4/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031555782Z namespaces/openshift-network-operator/pods/iptables-alerter-9psv4/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031686605Z namespaces/openshift-network-operator/pods/iptables-alerter-9psv4/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031769447Z namespaces/openshift-network-operator/pods/iptables-alerter-9psv4/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031805568Z namespaces/openshift-network-operator/pods/iptables-alerter-wnkrl/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031851289Z namespaces/openshift-network-operator/pods/iptables-alerter-wnkrl/iptables-alerter-wnkrl.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031946072Z namespaces/openshift-network-operator/pods/iptables-alerter-wnkrl/iptables-alerter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031959382Z namespaces/openshift-network-operator/pods/iptables-alerter-wnkrl/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031963572Z namespaces/openshift-network-operator/pods/iptables-alerter-wnkrl/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.031983832Z namespaces/openshift-network-operator/pods/iptables-alerter-wnkrl/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.032110445Z namespaces/openshift-network-operator/pods/iptables-alerter-wnkrl/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.032178987Z namespaces/openshift-network-operator/pods/iptables-alerter-wnkrl/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.032218778Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.032261099Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.03230852Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.032356442Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.032396112Z namespaces/openshift-node/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.032407873Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.032414763Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.032452394Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.032618118Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.032798932Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.033057189Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.033260804Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.03350591Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.033707375Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.033750496Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.033818358Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.033863439Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.03391105Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.033980382Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.034060573Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.034148386Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.034239858Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.03432487Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.034366821Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.034426653Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.034453913Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.034575446Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.034622507Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.034669908Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.034756061Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.034801942Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.034852463Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.034969816Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.035059868Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.03515844Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.035246803Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.035333575Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.035464598Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.035667723Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.035732705Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.035774435Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.035840447Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.035880128Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.03595211Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.035989501Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.036078583Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.036160665Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.036210366Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.036256337Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.036305579Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.03635972Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.036413341Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.036460563Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.036531514Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.036537634Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.036578435Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.036740029Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.037041987Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.03719345Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.037402926Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.037644772Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.037842837Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.037885898Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.03796915Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.0379888Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.038066882Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.038108863Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.038157364Z namespaces/openshift-operators/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.038166374Z namespaces/openshift-operators/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.038169885Z namespaces/openshift-operators/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.038206386Z namespaces/openshift-operators/coordination.k8s.io/leases/sail-operator-lock.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.038264127Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.038272027Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.038304348Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.038478112Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.038659026Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.03879725Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.039039686Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.039289962Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.039516088Z namespaces/openshift-operators/operators.coreos.com/installplans/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.039557889Z namespaces/openshift-operators/operators.coreos.com/installplans/install-kc5b2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.039930558Z namespaces/openshift-operators/operators.coreos.com/installplans/install-mlc5w.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.040290277Z namespaces/openshift-operators/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.040334288Z namespaces/openshift-operators/operators.coreos.com/operatorconditions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.040396649Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.04043271Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.040496462Z namespaces/openshift-operators/operators.coreos.com/subscriptions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.040526323Z namespaces/openshift-operators/operators.coreos.com/subscriptions/servicemeshoperator3.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.040604904Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.040644395Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.040699467Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.040755278Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.040802909Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.04084346Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041075206Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041155868Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.04124268Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041288551Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041332822Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041462206Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041503887Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041559628Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041600419Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041692111Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041737182Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041786454Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041869625Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041951427Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041963678Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.041998119Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.04205188Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.042100531Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.042322597Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.042415239Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.042736397Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.04286828Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.043374553Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.043469885Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.043630359Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.04368914Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.043734001Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.043828354Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.043965547Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.044009388Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.04410432Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.044156472Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.044194363Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.044261524Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.044310096Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.044373367Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.044414758Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.044464239Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.044470829Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.04451373Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.044568832Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.044614093Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.044694545Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.044784017Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.044831168Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.04488282Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.045041943Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.045077234Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.045172587Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.045212248Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.045269409Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.045275899Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.04531258Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.045403202Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.045454114Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.045498795Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.045545596Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.045551996Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.045593377Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.045742821Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.045940565Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.046151021Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.046362866Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.046618342Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.046829278Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.046841968Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.046884659Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovnkube-node-b27s8.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047065063Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/kube-rbac-proxy-node/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047075644Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047080164Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047112974Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047253508Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.04733114Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047368071Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047374381Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047377841Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047416142Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047558285Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047634207Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047674948Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/nbdb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047681448Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/nbdb/nbdb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047684919Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/nbdb/nbdb/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047727049Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/nbdb/nbdb/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047831392Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047921874Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047969695Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/northd/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047976656Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/northd/northd/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.047979936Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/northd/northd/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.048009407Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/northd/northd/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.04813017Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.048199661Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/northd/northd/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.048239982Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovn-acl-logging/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.048246622Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.048250122Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.048288043Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.048685993Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.048761005Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.048792316Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovn-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.048802606Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovn-controller/ovn-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.048814606Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.048849597Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.049197016Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.049285458Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.049306498Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovnkube-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.04935792Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.04936657Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.049410351Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.053771268Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.053884351Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.053950553Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/sbdb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.053967973Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/sbdb/sbdb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.053974713Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/sbdb/sbdb/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.053983423Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/sbdb/sbdb/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.054104866Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.054182848Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b27s8/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.054228749Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.054303941Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovnkube-node-p7h4b.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.054483436Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/kube-rbac-proxy-node/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.054492596Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.054497006Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.054531407Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.054697071Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.054773913Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.054814494Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.054822454Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.054831894Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.054855615Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055032659Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055110701Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055152702Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/nbdb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055160442Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/nbdb/nbdb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055164472Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/nbdb/nbdb/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055207084Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/nbdb/nbdb/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055307856Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055385198Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055422859Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/northd/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055429969Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/northd/northd/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055434119Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/northd/northd/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.05546667Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/northd/northd/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055582823Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055655324Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/northd/northd/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055692285Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovn-acl-logging/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055700165Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055704256Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.055743507Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.056197878Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.056254729Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovn-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.056261309Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovn-controller/ovn-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.05626483Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.056316841Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.056540546Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.056614668Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.056655289Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovnkube-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.056662049Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.056665609Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.056712061Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.061296633Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.061373985Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.061418646Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/sbdb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.061429117Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/sbdb/sbdb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.061433406Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/sbdb/sbdb/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.061462507Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/sbdb/sbdb/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.06157102Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.061650872Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-p7h4b/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.061681933Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.061759945Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovnkube-node-swcqx.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.061955139Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/kube-rbac-proxy-node/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.06196881Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.06197344Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062004291Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062167545Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062246947Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062288318Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062296728Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062302558Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062331329Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062476462Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062556034Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062597505Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/nbdb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062608406Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/nbdb/nbdb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062613856Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/nbdb/nbdb/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062651667Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/nbdb/nbdb/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062757719Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062830711Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062871242Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/northd/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062877622Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/northd/northd/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062880882Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/northd/northd/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.062947614Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/northd/northd/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.063063147Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.063138219Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/northd/northd/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.06317658Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovn-acl-logging/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.0631835Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.06318686Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.063226741Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.063635311Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.063693932Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovn-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.063700482Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovn-controller/ovn-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.063703882Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.063746843Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.06402606Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.064107622Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.064155804Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovnkube-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.064166494Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.064172134Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.064204375Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.068590093Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.068663224Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.068708985Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/sbdb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.068717246Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/sbdb/sbdb/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.068721406Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/sbdb/sbdb/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.068761337Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/sbdb/sbdb/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.068870329Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069013723Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-swcqx/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069040214Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069097825Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069165247Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069173077Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069207838Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.06930873Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069451774Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069502785Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069552356Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069645699Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069729801Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069781442Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069818583Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069870374Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069877344Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069880724Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.069957246Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.070017038Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.070023688Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.070075669Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.070273524Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.07050924Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.070657653Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.070882519Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.071171096Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.071432683Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.071478034Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.071598267Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.071644528Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.071691509Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.07174215Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.071833262Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.072017747Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.072109119Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.07215784Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.072196531Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.072252673Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.072294824Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.072382696Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.072428547Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.072478178Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.07255951Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.072610292Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.072618562Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.072660153Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.072712034Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.072759485Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.072870598Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.073001331Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.073197076Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.073278488Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.073400121Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.073526124Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.073710799Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.07377232Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.073814941Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.073863082Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.073929054Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.073988995Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.074036667Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.074119309Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.074207591Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.074264562Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.074308093Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.074362904Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.074403765Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.074453237Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.074511438Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.074554389Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.074560969Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.07459908Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.074757044Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.074964569Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.075105293Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.075309608Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.075549284Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.075752039Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.075760219Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-d6fc45fc5-fc8h5/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.0758003Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-d6fc45fc5-fc8h5/service-ca-operator-d6fc45fc5-fc8h5.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.075885472Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-d6fc45fc5-fc8h5/service-ca-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.075893392Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-d6fc45fc5-fc8h5/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.075897432Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-d6fc45fc5-fc8h5/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.075953724Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-d6fc45fc5-fc8h5/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.076121578Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-d6fc45fc5-fc8h5/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.076263981Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.076306102Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.076362354Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.076405405Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.076461646Z namespaces/openshift-service-ca/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.076506237Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.076564129Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.07660674Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.076660851Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.076703872Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.076801204Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.076896197Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.077072061Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.077111262Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.077165904Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.077216205Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.077261646Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.077341618Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.077391559Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.07743932Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.077520382Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.077567063Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.077574334Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.077608454Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.077678636Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.077713607Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.0778294Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.077913162Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.078050895Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.078125407Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.07825668Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.078376383Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.078585968Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.07864203Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.078685511Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.078736022Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.078781723Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.078830924Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.078881366Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.079006769Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.079096661Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.079145592Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.079194913Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.079240644Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.079284645Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.079334207Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.079341227Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.079381308Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.079539492Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.079715236Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.07985839Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.080087225Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.080334582Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.080538756Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.080547247Z namespaces/openshift-service-ca/pods/service-ca-865cb79987-x4ng7/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.080586658Z namespaces/openshift-service-ca/pods/service-ca-865cb79987-x4ng7/service-ca-865cb79987-x4ng7.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.08066175Z namespaces/openshift-service-ca/pods/service-ca-865cb79987-x4ng7/service-ca-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.08066987Z namespaces/openshift-service-ca/pods/service-ca-865cb79987-x4ng7/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.08067408Z namespaces/openshift-service-ca/pods/service-ca-865cb79987-x4ng7/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.08070334Z namespaces/openshift-service-ca/pods/service-ca-865cb79987-x4ng7/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.080932736Z namespaces/openshift-service-ca/pods/service-ca-865cb79987-x4ng7/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081043629Z namespaces/openshift-service-ca/pods/service-ca-865cb79987-x4ng7/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.08107846Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081116911Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081172842Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081223323Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081257094Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081317926Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081373597Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081420448Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081470689Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.08150793Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081599973Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081679815Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081770427Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081813958Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081862429Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081931071Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.081976582Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.082063344Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.082114485Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.082156086Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.082243288Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.082286659Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.082331931Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.082438893Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.082518815Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.082609377Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.08269316Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.082787262Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.082927125Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083101459Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083157711Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083198272Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083249043Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083296064Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083342145Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083394307Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083467748Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083557451Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083609612Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083658043Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083705994Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083752535Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083810147Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083822667Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.083866568Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.084058503Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.084249068Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.084394821Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.084588316Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.084831462Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085077078Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085126659Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085187831Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085217622Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085269503Z namespaces/openshift/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085314394Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085370375Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085413056Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085464518Z namespaces/openshift/apps/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085505469Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085603731Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085680723Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085765895Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085811716Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085860687Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.085923699Z namespaces/openshift/batch/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.0859756Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.086061552Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.086109304Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.086154714Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.086235456Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.086289978Z namespaces/openshift/core/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.086335489Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.086440232Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.086525704Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.086611326Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.086696738Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.08678371Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.086924013Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.087093808Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.087152149Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.08719971Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.087247772Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.087298823Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.088505282Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.088539913Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.088642736Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.088726398Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.088844041Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.089058566Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.089150348Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.089302852Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.089469936Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.089597859Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.089699052Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.089807394Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.089951858Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.090062711Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.090174513Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.090269266Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.090361518Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.090470571Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.090592564Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.090709487Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.090813919Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.090929642Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.091046285Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.091149628Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.0912533Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.091364273Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.091456415Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.091551337Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.0916619Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.091775083Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.091876165Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.092010329Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.092129832Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.092225424Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.092333906Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.092429259Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.092550402Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.092707676Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.092808468Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.092946622Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.093060264Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.093186447Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.093321491Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.093419293Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.093513426Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.093646019Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.093779702Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.093887655Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.094056909Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.094154161Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.094248844Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.094354916Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.094473129Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.094592792Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.094718535Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.094833398Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.094978451Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.095086414Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.095181727Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.09532785Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.095411902Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.095458123Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.095551666Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.095635298Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.095687769Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.09573584Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.095781621Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.095829732Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.095877574Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.095883754Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.095946665Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.096101029Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.096280763Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.096426127Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.3.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.096626132Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.3.2.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.096878648Z namespaces/openshift/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.1.0.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.097109354Z namespaces/openshift/policy/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.097156025Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.097217326Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.097254218Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.097287388Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.097300249Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.09734126Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.097491143Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.097618996Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.097727979Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.097847242Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.098045417Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.09815275Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.098273303Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.098384175Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.098512638Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.098634941Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.098753904Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.098884138Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.099105743Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.099211236Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.099340749Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.099452892Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.099576295Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.099687537Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.09980339Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.099918563Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.100041106Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.100151419Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.100273752Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.100370884Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.100476207Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.100584469Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.100699512Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.100812785Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.100946128Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.101082952Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.101181554Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.101289237Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.10141667Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.101534183Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.101643966Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.101758818Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.101866451Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.102011555Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.102126247Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.10223452Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.102375333Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.102494746Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.10264403Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.102772373Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.102920087Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.103077051Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.103210944Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.103345927Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.10347024Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.103599494Z network_logs/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.103645025Z network_logs/cluster_scale [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.103735807Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.103817619Z network_logs/multi-networkpolicy [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.103920622Z network_logs/net-attach-def [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.104022444Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.104170338Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.10427009Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.106035943Z nodes/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.106057304Z nodes/debug [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.106169617Z nodes/ip-10-0-137-68.ec2.internal/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.106210158Z nodes/ip-10-0-137-68.ec2.internal/cpu_affinities.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.106484365Z nodes/ip-10-0-137-68.ec2.internal/dmesg [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.106766491Z nodes/ip-10-0-137-68.ec2.internal/ethtool_channels [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.106848614Z nodes/ip-10-0-137-68.ec2.internal/ethtool_features [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.107004487Z nodes/ip-10-0-137-68.ec2.internal/ip-10-0-137-68.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.107370336Z nodes/ip-10-0-137-68.ec2.internal/irq_affinities.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.10752645Z nodes/ip-10-0-137-68.ec2.internal/lscpu [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.107661154Z nodes/ip-10-0-137-68.ec2.internal/lspci [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.107765096Z nodes/ip-10-0-137-68.ec2.internal/podresources.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.107872309Z nodes/ip-10-0-137-68.ec2.internal/pods_info.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.108018232Z nodes/ip-10-0-137-68.ec2.internal/proc_cmdline [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.108110754Z nodes/ip-10-0-137-68.ec2.internal/sysinfo.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.109777655Z nodes/ip-10-0-137-68.ec2.internal/sysinfo.tgz [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.1099621Z nodes/ip-10-0-138-68.ec2.internal/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.110003991Z nodes/ip-10-0-138-68.ec2.internal/cpu_affinities.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.110335909Z nodes/ip-10-0-138-68.ec2.internal/dmesg [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.110597636Z nodes/ip-10-0-138-68.ec2.internal/ethtool_channels [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.110681118Z nodes/ip-10-0-138-68.ec2.internal/ethtool_features [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.11076989Z nodes/ip-10-0-138-68.ec2.internal/ip-10-0-138-68.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.11117017Z nodes/ip-10-0-138-68.ec2.internal/irq_affinities.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.111328654Z nodes/ip-10-0-138-68.ec2.internal/lscpu [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.111451637Z nodes/ip-10-0-138-68.ec2.internal/lspci [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.111550759Z nodes/ip-10-0-138-68.ec2.internal/podresources.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.111644971Z nodes/ip-10-0-138-68.ec2.internal/pods_info.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.111778875Z nodes/ip-10-0-138-68.ec2.internal/proc_cmdline [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.111869127Z nodes/ip-10-0-138-68.ec2.internal/sysinfo.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.113565889Z nodes/ip-10-0-138-68.ec2.internal/sysinfo.tgz [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.113719052Z nodes/ip-10-0-143-218.ec2.internal/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.113762533Z nodes/ip-10-0-143-218.ec2.internal/cpu_affinities.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.114095302Z nodes/ip-10-0-143-218.ec2.internal/dmesg [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.114337008Z nodes/ip-10-0-143-218.ec2.internal/ethtool_channels [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.11442Z nodes/ip-10-0-143-218.ec2.internal/ethtool_features [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.114521002Z nodes/ip-10-0-143-218.ec2.internal/ip-10-0-143-218.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.11485777Z nodes/ip-10-0-143-218.ec2.internal/irq_affinities.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.115109827Z nodes/ip-10-0-143-218.ec2.internal/lscpu [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.11525409Z nodes/ip-10-0-143-218.ec2.internal/lspci [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.115346112Z nodes/ip-10-0-143-218.ec2.internal/podresources.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.115461555Z nodes/ip-10-0-143-218.ec2.internal/pods_info.json [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.115593218Z nodes/ip-10-0-143-218.ec2.internal/proc_cmdline [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.115694691Z nodes/ip-10-0-143-218.ec2.internal/sysinfo.log [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.117382213Z nodes/ip-10-0-143-218.ec2.internal/sysinfo.tgz [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.117554357Z pod_network_connectivity_check/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.117606128Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.117718321Z static-pods/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.117725881Z static-pods/kube-apiserver/ [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.124803445Z [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.124822225Z sent 70,315 bytes received 6,997,775 bytes 4,712,060.00 bytes/sec [must-gather] [must-gather-gr2p9] OUT 2026-04-23T19:17:34.124826896Z total size is 82,860,297 speedup is 11.72 [must-gather] [must-gather ] OUT 2026-04-23T19:17:34.416682463Z namespace/openshift-must-gather-j4sk6 deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 4070c745-bd54-4f25-90de-26000850e63f [must-gather] ClientVersion: 4.21.5 [must-gather] ClusterVersion: Stable at "4.20.19" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-hprf8 [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-hprf8-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-hprf8 [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 15082976 Apr 23 19:18 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-hprf8/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts 206a3ad] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-hprf8/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] cf81b50..206a3ad ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-04-23T19:19:00.745Z","type":3}]