task e2e-predictor has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-2lmjn-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-2lmjn-kubeconfig [get-kubeconfig] Found admin password secret: cluster-2lmjn-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-2lmjn-password [get-kubeconfig] API Server URL: https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [clone-repo] work/RHOAIENG-79440 [clone-repo] https://github.com/jlost/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 85% (2793/3251) Updating files: 86% (2796/3251) Updating files: 87% (2829/3251) Updating files: 88% (2861/3251) Updating files: 89% (2894/3251) Updating files: 90% (2926/3251) Updating files: 91% (2959/3251) Updating files: 92% (2991/3251) Updating files: 93% (3024/3251) Updating files: 94% (3056/3251) Updating files: 95% (3089/3251) Updating files: 96% (3121/3251) Updating files: 97% (3154/3251) Updating files: 98% (3186/3251) Updating files: 99% (3219/3251) Updating files: 100% (3251/3251) Updating files: 100% (3251/3251), done. [e2e-predictor] + bash [e2e-predictor] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-predictor] + echo failed [e2e-predictor] + COMPONENT_NAME=kserve-agent-ci [e2e-predictor] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-predictor] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:440184469bf83caf5543427fdbbf80873e43c9c7af817ffabe7c58eda76f78c2 [e2e-predictor] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:440184469bf83caf5543427fdbbf80873e43c9c7af817ffabe7c58eda76f78c2 [e2e-predictor] + COMPONENT_NAME=kserve-controller-ci [e2e-predictor] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-predictor] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:a627ec30debc454af106c2c34af695e96eccb25d564ea70da2ca083e0bc41d91 [e2e-predictor] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:a627ec30debc454af106c2c34af695e96eccb25d564ea70da2ca083e0bc41d91 [e2e-predictor] + COMPONENT_NAME=kserve-router-ci [e2e-predictor] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-predictor] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:ecd8623c42b521978afd08b50dfd4d13bfe521fd8131429e4f24493619574031 [e2e-predictor] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:ecd8623c42b521978afd08b50dfd4d13bfe521fd8131429e4f24493619574031 [e2e-predictor] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-predictor] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-predictor] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:0a55b7ab9c6c77d762de859d20496ef5a9b41cb492d5728752f25851b1e70916 [e2e-predictor] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:0a55b7ab9c6c77d762de859d20496ef5a9b41cb492d5728752f25851b1e70916 [e2e-predictor] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'predictor or kserve_on_openshift' [e2e-predictor] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.26) [e2e-predictor] Installing on cluster [e2e-predictor] Using namespace: kserve for KServe components [e2e-predictor] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821 [e2e-predictor] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-predictor] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1821 [e2e-predictor] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1821 [e2e-predictor] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-predictor] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-predictor] v5.8.1 [e2e-predictor] make: Entering directory '/workspace/source' [e2e-predictor] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-predictor] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-predictor] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-predictor] make: Leaving directory '/workspace/source' [e2e-predictor] Installing KServe Python SDK ... [e2e-predictor] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-predictor] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-predictor] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-predictor] uv 0.7.8 [e2e-predictor] Creating virtual environment... [e2e-predictor] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-predictor] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-predictor] Creating virtual environment at: .venv [e2e-predictor] /workspace/source [e2e-predictor] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-predictor] Creating virtual environment at: .venv [e2e-predictor] Resolved 274 packages in 1ms [e2e-predictor] Building kserve @ file:///workspace/source/python/kserve [e2e-predictor] Downloading aiohttp (1.7MiB) [e2e-predictor] Downloading pydantic-core (2.0MiB) [e2e-predictor] Downloading kubernetes (1.9MiB) [e2e-predictor] Downloading uvloop (3.8MiB) [e2e-predictor] Downloading cryptography (4.5MiB) [e2e-predictor] Downloading pyarrow (40.1MiB) [e2e-predictor] Downloading black (1.6MiB) [e2e-predictor] Downloading botocore (12.9MiB) [e2e-predictor] Downloading grpcio (6.4MiB) [e2e-predictor] Downloading pandas (12.5MiB) [e2e-predictor] Downloading mypy (17.2MiB) [e2e-predictor] Downloading portforward (3.9MiB) [e2e-predictor] Downloading setuptools (1.2MiB) [e2e-predictor] Downloading grpcio-tools (2.5MiB) [e2e-predictor] Downloading numpy (15.7MiB) [e2e-predictor] Building timeout-sampler==1.0.3 [e2e-predictor] Building python-simple-logger==2.0.19 [e2e-predictor] Downloading aiohttp [e2e-predictor] Downloading black [e2e-predictor] Downloading pydantic-core [e2e-predictor] Downloading grpcio-tools [e2e-predictor] Downloading setuptools [e2e-predictor] Built python-simple-logger==2.0.19 [e2e-predictor] Downloading portforward [e2e-predictor] Downloading uvloop [e2e-predictor] Downloading cryptography [e2e-predictor] Downloading grpcio [e2e-predictor] Downloading kubernetes [e2e-predictor] Built timeout-sampler==1.0.3 [e2e-predictor] Built kserve @ file:///workspace/source/python/kserve [e2e-predictor] Downloading numpy [e2e-predictor] Downloading pandas [e2e-predictor] Downloading botocore [e2e-predictor] Downloading pyarrow [e2e-predictor] Downloading mypy [e2e-predictor] Prepared 101 packages in 1.87s [e2e-predictor] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-predictor] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-predictor] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-predictor] Installed 101 packages in 4.92s [e2e-predictor] + aiohappyeyeballs==2.6.1 [e2e-predictor] + aiohttp==3.14.1 [e2e-predictor] + aiosignal==1.4.0 [e2e-predictor] + annotated-doc==0.0.4 [e2e-predictor] + annotated-types==0.7.0 [e2e-predictor] + anyio==4.9.0 [e2e-predictor] + attrs==25.3.0 [e2e-predictor] + avro==1.12.0 [e2e-predictor] + black==24.3.0 [e2e-predictor] + boto3==1.37.35 [e2e-predictor] + botocore==1.37.35 [e2e-predictor] + cachetools==5.5.2 [e2e-predictor] + certifi==2025.1.31 [e2e-predictor] + cffi==2.0.0 [e2e-predictor] + charset-normalizer==3.4.1 [e2e-predictor] + click==8.4.2 [e2e-predictor] + cloudevents==1.11.0 [e2e-predictor] + colorama==0.4.6 [e2e-predictor] + colorlog==6.10.1 [e2e-predictor] + coverage==7.8.0 [e2e-predictor] + cryptography==49.0.0 [e2e-predictor] + deprecation==2.1.0 [e2e-predictor] + durationpy==0.9 [e2e-predictor] + execnet==2.1.1 [e2e-predictor] + fastapi==0.136.3 [e2e-predictor] + frozenlist==1.5.0 [e2e-predictor] + google-auth==2.39.0 [e2e-predictor] + grpc-interceptor==0.15.4 [e2e-predictor] + grpcio==1.78.1 [e2e-predictor] + grpcio-testing==1.78.1 [e2e-predictor] + grpcio-tools==1.78.1 [e2e-predictor] + h11==0.16.0 [e2e-predictor] + httpcore==1.0.9 [e2e-predictor] + httptools==0.6.4 [e2e-predictor] + httpx==0.27.2 [e2e-predictor] + httpx-retries==0.4.5 [e2e-predictor] + idna==3.10 [e2e-predictor] + iniconfig==2.1.0 [e2e-predictor] + jinja2==3.1.6 [e2e-predictor] + jmespath==1.0.1 [e2e-predictor] + kserve==0.20.0rc0 (from file:///workspace/source/python/kserve) [e2e-predictor] + kubernetes==32.0.1 [e2e-predictor] + markupsafe==3.0.2 [e2e-predictor] + multidict==6.4.3 [e2e-predictor] + mypy==0.991 [e2e-predictor] + mypy-extensions==1.0.0 [e2e-predictor] + numpy==2.2.4 [e2e-predictor] + oauthlib==3.2.2 [e2e-predictor] + orjson==3.11.9 [e2e-predictor] + packaging==24.2 [e2e-predictor] + pandas==2.2.3 [e2e-predictor] + pathspec==0.12.1 [e2e-predictor] + platformdirs==4.3.7 [e2e-predictor] + pluggy==1.5.0 [e2e-predictor] + portforward==0.7.1 [e2e-predictor] + prometheus-client==0.21.1 [e2e-predictor] + propcache==0.3.1 [e2e-predictor] + protobuf==6.33.5 [e2e-predictor] + psutil==5.9.8 [e2e-predictor] + pyarrow==19.0.1 [e2e-predictor] + pyasn1==0.6.3 [e2e-predictor] + pyasn1-modules==0.4.2 [e2e-predictor] + pycparser==2.22 [e2e-predictor] + pydantic==2.12.4 [e2e-predictor] + pydantic-core==2.41.5 [e2e-predictor] + pyjwt==2.13.0 [e2e-predictor] + pytest==7.4.4 [e2e-predictor] + pytest-asyncio==0.23.8 [e2e-predictor] + pytest-cov==5.0.0 [e2e-predictor] + pytest-httpx==0.30.0 [e2e-predictor] + pytest-json-report==1.5.0 [e2e-predictor] + pytest-metadata==3.1.1 [e2e-predictor] + pytest-xdist==3.6.1 [e2e-predictor] + python-dateutil==2.9.0.post0 [e2e-predictor] + python-dotenv==1.1.0 [e2e-predictor] + python-multipart==0.0.32 [e2e-predictor] + python-simple-logger==2.0.19 [e2e-predictor] + pytz==2025.2 [e2e-predictor] + pyyaml==6.0.2 [e2e-predictor] + requests==2.32.3 [e2e-predictor] + requests-oauthlib==2.0.0 [e2e-predictor] + rsa==4.9.1 [e2e-predictor] + s3transfer==0.11.4 [e2e-predictor] + setuptools==78.1.0 [e2e-predictor] + six==1.17.0 [e2e-predictor] + sniffio==1.3.1 [e2e-predictor] + starlette==1.2.1 [e2e-predictor] + tabulate==0.9.0 [e2e-predictor] + timeout-sampler==1.0.3 [e2e-predictor] + timing-asgi==0.3.1 [e2e-predictor] + tomlkit==0.13.2 [e2e-predictor] + typing-extensions==4.15.0 [e2e-predictor] + typing-inspection==0.4.2 [e2e-predictor] + tzdata==2025.2 [e2e-predictor] + urllib3==2.7.0 [e2e-predictor] + uvicorn==0.34.1 [e2e-predictor] + uvloop==0.21.0 [e2e-predictor] + watchfiles==1.0.5 [e2e-predictor] + websocket-client==1.8.0 [e2e-predictor] + websockets==15.0.1 [e2e-predictor] + yarl==1.20.0 [e2e-predictor] Audited 1 package in 47ms [e2e-predictor] /workspace/source [e2e-predictor] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-predictor] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-predictor] make: Entering directory '/workspace/source' [e2e-predictor] make: Leaving directory '/workspace/source' [e2e-predictor] Now using project "kserve" on server "https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443". [e2e-predictor] [e2e-predictor] You can add applications to this project with the 'new-app' command. For example, try: [e2e-predictor] [e2e-predictor] oc new-app rails-postgresql-example [e2e-predictor] [e2e-predictor] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-predictor] [e2e-predictor] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-predictor] [e2e-predictor] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-predictor] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-predictor] make: Entering directory '/workspace/source' [e2e-predictor] make: Leaving directory '/workspace/source' [e2e-predictor] Creating namespace openshift-keda... [e2e-predictor] namespace/openshift-keda created [e2e-predictor] Namespace openshift-keda created/ensured. [e2e-predictor] --- [e2e-predictor] Creating OperatorGroup openshift-keda... [e2e-predictor] operatorgroup.operators.coreos.com/openshift-keda created [e2e-predictor] OperatorGroup openshift-keda created/ensured. [e2e-predictor] --- [e2e-predictor] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-predictor] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-predictor] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-predictor] --- [e2e-predictor] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-predictor] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-predictor] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-predictor] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-predictor] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-predictor] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-predictor] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (25/600) [e2e-predictor] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-predictor] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-predictor] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-predictor] CSV custom-metrics-autoscaler.v2.19.0-2 is ready (Phase: Succeeded). [e2e-predictor] --- [e2e-predictor] Applying KedaController custom resource... [e2e-predictor] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-predictor] kedacontroller.keda.sh/keda configured [e2e-predictor] KedaController custom resource applied. [e2e-predictor] --- [e2e-predictor] Allowing time for KEDA components to be provisioned by the operator ... [e2e-predictor] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-predictor] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-predictor] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-predictor] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-predictor] NAME READY STATUS RESTARTS AGE [e2e-predictor] keda-operator-77f648d694-bzvhk 1/1 Running 0 52s [e2e-predictor] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-predictor] pod/keda-operator-77f648d694-bzvhk condition met [e2e-predictor] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-predictor] KEDA Operator pod is ready. [e2e-predictor] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-predictor] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-predictor] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-predictor] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-predictor] NAME READY STATUS RESTARTS AGE [e2e-predictor] keda-metrics-apiserver-55557df8c-79p8k 1/1 Running 0 66s [e2e-predictor] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-predictor] pod/keda-metrics-apiserver-55557df8c-79p8k condition met [e2e-predictor] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-predictor] KEDA Metrics API Server pod is ready. [e2e-predictor] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-predictor] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-predictor] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-predictor] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-predictor] NAME READY STATUS RESTARTS AGE [e2e-predictor] keda-admission-56f9798bb4-l6nwm 1/1 Running 0 82s [e2e-predictor] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-predictor] pod/keda-admission-56f9798bb4-l6nwm condition met [e2e-predictor] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-predictor] KEDA Webhook pod is ready. [e2e-predictor] --- [e2e-predictor] ✅ KEDA deployment script finished successfully. [e2e-predictor] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:a627ec30debc454af106c2c34af695e96eccb25d564ea70da2ca083e0bc41d91 [e2e-predictor] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller:odh-master [e2e-predictor] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:440184469bf83caf5543427fdbbf80873e43c9c7af817ffabe7c58eda76f78c2 [e2e-predictor] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:ecd8623c42b521978afd08b50dfd4d13bfe521fd8131429e4f24493619574031 [e2e-predictor] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:0a55b7ab9c6c77d762de859d20496ef5a9b41cb492d5728752f25851b1e70916 [e2e-predictor] Installing KServe via kustomize... [e2e-predictor] === Final params.env [e2e-predictor] kserve-controller=quay.io/opendatahub/kserve-controller@sha256:a627ec30debc454af106c2c34af695e96eccb25d564ea70da2ca083e0bc41d91 [e2e-predictor] llmisvc-controller=quay.io/opendatahub/odh-kserve-llmisvc-controller:odh-master [e2e-predictor] kserve-agent=quay.io/opendatahub/kserve-agent@sha256:440184469bf83caf5543427fdbbf80873e43c9c7af817ffabe7c58eda76f78c2 [e2e-predictor] kserve-router=quay.io/opendatahub/kserve-router@sha256:ecd8623c42b521978afd08b50dfd4d13bfe521fd8131429e4f24493619574031 [e2e-predictor] kserve-storage-initializer=quay.io/opendatahub/kserve-storage-initializer@sha256:0a55b7ab9c6c77d762de859d20496ef5a9b41cb492d5728752f25851b1e70916 [e2e-predictor] kserve-llm-d=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-predictor] kserve-llm-d-inference-scheduler=quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-predictor] kserve-llm-d-routing-sidecar=quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-predictor] kserve-llm-d-uds-tokenizer=quay.io/opendatahub/llm-d-kv-cache:v0.8.0 [e2e-predictor] kserve-llm-d-nvidia-cuda=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-predictor] kserve-llm-d-nvidia-cuda-fast-1=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-predictor] kserve-llm-d-nvidia-cuda-fast-2=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-predictor] kserve-llm-d-nvidia-cuda-upstream-version=0.11.0+rhai5 [e2e-predictor] kserve-llm-d-nvidia-cuda-fast-1-upstream-version=0.11.0+rhai5 [e2e-predictor] kserve-llm-d-nvidia-cuda-fast-2-upstream-version=0.11.0+rhai5 [e2e-predictor] kserve-llm-d-amd-rocm=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-predictor] kserve-llm-d-amd-rocm-fast-1=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-predictor] kserve-llm-d-amd-rocm-fast-2=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-predictor] kserve-llm-d-amd-rocm-upstream-version=0.11.0+rhai5 [e2e-predictor] kserve-llm-d-amd-rocm-fast-1-upstream-version=0.11.0+rhai5 [e2e-predictor] kserve-llm-d-amd-rocm-fast-2-upstream-version=0.11.0+rhai5 [e2e-predictor] kserve-llm-d-intel-gaudi=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-predictor] kserve-llm-d-intel-gaudi-fast-1=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-predictor] kserve-llm-d-intel-gaudi-fast-2=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-predictor] kserve-llm-d-intel-gaudi-upstream-version=0.16.0 [e2e-predictor] kserve-llm-d-intel-gaudi-fast-1-upstream-version=0.16.0 [e2e-predictor] kserve-llm-d-intel-gaudi-fast-2-upstream-version=0.16.0 [e2e-predictor] kserve-llm-d-ibm-spyre=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-predictor] kserve-llm-d-ibm-spyre-fast-1=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-predictor] kserve-llm-d-ibm-spyre-fast-2=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-predictor] kserve-llm-d-ibm-spyre-upstream-version=0.10.2 [e2e-predictor] kserve-llm-d-ibm-spyre-fast-1-upstream-version=0.10.2 [e2e-predictor] kserve-llm-d-ibm-spyre-fast-2-upstream-version=0.10.2 [e2e-predictor] kserve-llm-d-latency-predictor-prediction=quay.io/opendatahub/odh-latency-predictor-prediction:odh-stable [e2e-predictor] kserve-llm-d-latency-predictor-training=quay.io/opendatahub/odh-latency-predictor-training:odh-stable [e2e-predictor] # TODO update when our changes are introduced in the official image [e2e-predictor] kube-rbac-proxy=quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3 [e2e-predictor] kserve-localmodel-controller=quay.io/opendatahub/odh-kserve-localmodel-controller:odh-master [e2e-predictor] kserve-localmodelnode-agent=quay.io/opendatahub/odh-kserve-localmodelnode-agent:odh-master [e2e-predictor] ovms-versioning-ubi-micro=registry.redhat.io/ubi9/ubi-micro@sha256:38e934147827349f2b8b11ac9c38d7be23cb8e29f128b1c46e5c7ae54a2d23cd [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-predictor] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-predictor] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-predictor] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-predictor] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-predictor] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-predictor] Waiting for CRDs to be established... [e2e-predictor] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-predictor] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-predictor] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-predictor] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-predictor] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-predictor] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-predictor] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-predictor] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-predictor] Applying all resources... [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-predictor] serviceaccount/kserve-controller-manager serverside-applied [e2e-predictor] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-predictor] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-predictor] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-predictor] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-predictor] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-predictor] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-predictor] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-predictor] configmap/inferenceservice-config serverside-applied [e2e-predictor] configmap/kserve-parameters serverside-applied [e2e-predictor] secret/kserve-webhook-server-secret serverside-applied [e2e-predictor] secret/mlpipeline-s3-artifact serverside-applied [e2e-predictor] service/kserve-controller-manager-metrics-service serverside-applied [e2e-predictor] service/kserve-controller-manager-service serverside-applied [e2e-predictor] service/kserve-webhook-server-service serverside-applied [e2e-predictor] service/llmisvc-controller-manager-service serverside-applied [e2e-predictor] service/llmisvc-webhook-server-service serverside-applied [e2e-predictor] service/s3-service serverside-applied [e2e-predictor] deployment.apps/kserve-controller-manager serverside-applied [e2e-predictor] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-predictor] deployment.apps/seaweedfs serverside-applied [e2e-predictor] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-predictor] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-predictor] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-predictor] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-predictor] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-predictor] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-predictor] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-predictor] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-predictor] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-predictor] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-predictor] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-predictor] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-predictor] Waiting for llmisvc-controller-manager to be ready... [e2e-predictor] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-predictor] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-predictor] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-predictor] NAME READY STATUS RESTARTS AGE [e2e-predictor] llmisvc-controller-manager-654c557856-rhl7b 0/1 ContainerCreating 0 6s [e2e-predictor] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-predictor] pod/llmisvc-controller-manager-654c557856-rhl7b condition met [e2e-predictor] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-predictor] Re-Applying all resources... [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-predictor] serviceaccount/kserve-controller-manager serverside-applied [e2e-predictor] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-predictor] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-predictor] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-predictor] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-predictor] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-predictor] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-predictor] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-predictor] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-predictor] configmap/inferenceservice-config serverside-applied [e2e-predictor] configmap/kserve-parameters serverside-applied [e2e-predictor] secret/kserve-webhook-server-secret serverside-applied [e2e-predictor] secret/mlpipeline-s3-artifact serverside-applied [e2e-predictor] service/kserve-controller-manager-metrics-service serverside-applied [e2e-predictor] service/kserve-controller-manager-service serverside-applied [e2e-predictor] service/kserve-webhook-server-service serverside-applied [e2e-predictor] service/llmisvc-controller-manager-service serverside-applied [e2e-predictor] service/llmisvc-webhook-server-service serverside-applied [e2e-predictor] service/s3-service serverside-applied [e2e-predictor] deployment.apps/kserve-controller-manager serverside-applied [e2e-predictor] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-predictor] deployment.apps/seaweedfs serverside-applied [e2e-predictor] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-predictor] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-predictor] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-predictor] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-predictor] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-predictor] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-predictor] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-predictor] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-predictor] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-predictor] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-predictor] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-predictor] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-predictor] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-predictor] Applying DSC/DSCI resources... [e2e-predictor] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-predictor] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-predictor] KServe manual installation complete [e2e-predictor] Patching ingress domain... [e2e-predictor] configmap/inferenceservice-config patched [e2e-predictor] pod "kserve-controller-manager-66f44fdc75-q8nnc" deleted [e2e-predictor] Waiting for kserve-controller-manager to be ready... [e2e-predictor] pod/kserve-controller-manager-66f44fdc75-5jw6z condition met [e2e-predictor] Installing ODH Model Controller manually... [e2e-predictor] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-predictor] serviceaccount/model-serving-api created [e2e-predictor] serviceaccount/odh-model-controller created [e2e-predictor] role.rbac.authorization.k8s.io/leader-election-role created [e2e-predictor] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-predictor] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-predictor] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-predictor] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-predictor] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-predictor] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-predictor] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-predictor] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-predictor] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-predictor] configmap/odh-model-controller-parameters created [e2e-predictor] service/model-serving-api created [e2e-predictor] service/odh-model-controller-metrics-service created [e2e-predictor] service/odh-model-controller-webhook-service created [e2e-predictor] deployment.apps/model-serving-api created [e2e-predictor] deployment.apps/odh-model-controller created [e2e-predictor] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-predictor] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-predictor] template.template.openshift.io/autogluon-runtime-template created [e2e-predictor] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-predictor] template.template.openshift.io/kserve-ovms created [e2e-predictor] template.template.openshift.io/mlserver-cuda-runtime-template created [e2e-predictor] template.template.openshift.io/mlserver-runtime-template created [e2e-predictor] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-predictor] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-predictor] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-predictor] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-predictor] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-predictor] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-predictor] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-predictor] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-predictor] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-predictor] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-predictor] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-predictor] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-predictor] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-predictor] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-predictor] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-predictor] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-predictor] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-predictor] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-predictor] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-predictor] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-predictor] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-predictor] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-predictor] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-predictor] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-predictor] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-predictor] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-predictor] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-predictor] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-predictor] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-predictor] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-predictor] deployment "odh-model-controller" successfully rolled out [e2e-predictor] networkpolicy.networking.k8s.io/allow-all created [e2e-predictor] KServe setup complete (namespace: kserve) [e2e-predictor] Add testing models to SeaweedFS S3 storage ... [e2e-predictor] Waiting for SeaweedFS deployment to be ready... [e2e-predictor] deployment "seaweedfs" successfully rolled out [e2e-predictor] S3 init job not completed, re-creating... [e2e-predictor] job.batch/s3-init replaced [e2e-predictor] Waiting for S3 init job to complete... [e2e-predictor] job.batch/s3-init condition met [e2e-predictor] Configuring SeaweedFS S3 TLS [e2e-predictor] Namespace kserve exists. [e2e-predictor] Cleaning up existing custom TLS SeaweedFS resources for idempotency... [e2e-predictor] secret/seaweedfs-tls-custom-artifact serverside-applied [e2e-predictor] service/seaweedfs-tls-custom-service serverside-applied [e2e-predictor] deployment.apps/seaweedfs-tls-custom serverside-applied [e2e-predictor] Waiting for seaweedfs-tls-custom deployment to be ready... [e2e-predictor] Waiting for deployment "seaweedfs-tls-custom" rollout to finish: 0 of 1 updated replicas are available... [e2e-predictor] deployment "seaweedfs-tls-custom" successfully rolled out [e2e-predictor] Configuring SeaweedFS S3 for TLS with custom certificate and adding models to storage [e2e-predictor] Generating Custom CA cert and secret [e2e-predictor] ......+.......+............+...+.....+.+......+......+..............+.+++++++++++++++++++++++++++++++++++++++++++++*...+.....+.+++++++++++++++++++++++++++++++++++++++++++++*.+.+........+......+....+..+.........+......+.......+.................+.............+...........+.......+........+.+...+...+..+......+.+......+...............+..................+..+...+....+...+..+.+.........+..+......+...+.+...........+.........+....+...........+....+++++ [e2e-predictor] .....+...............+.....+.+...+...........+.+..+......+....+........+.........+.+++++++++++++++++++++++++++++++++++++++++++++*....+.+...+.....+.......+..............+++++++++++++++++++++++++++++++++++++++++++++*...............+......+...............+.+............+..+...+......+.+........+.+.....+................+.....+......+...+.......+..+.............+.................................+..+.......+.....+.+.....+......+.+...+......+........+.+...........+.+....................+............+.......+++++ [e2e-predictor] ----- [e2e-predictor] .+.+........+...+.......+..+++++++++++++++++++++++++++++++++++++++++++++*....+.+.....+..........+........+...+...+....+..+.............+...+..+++++++++++++++++++++++++++++++++++++++++++++*.....+...+........+......+..........+......+....................+.......+...+............+..+.+...+..+.......+..+.........+.+......+.....+............+.......+.........+.....+.+......+..+...+.......+..+.........+......+..........+...+........+...+....+...+.................+......+.......+...+..+.......+...+.....+.......+..+......+...+..........+......+...............+.........+.........+......+.....+......+...+............+......+...+.+.....................+............+..+...+.............+...+..............+............+.+..+.+......+.....+.+.........+.........+..+.......+...+.................+.+.....+............+...+.+.....+.+........+...............+....+...........+....+..+......+...+....+.....+...+.......+.......................+....+...............+........+.+.....+....+.................+....+.........+..+...+...+.......+......+..+....+........+.+..+.............+............+.................+......+......+....+............+..+.+......+........+...+.......+........+............+......+.+.............................+.+..................+...+.........+...........+.............+....................+......+..........+.....+.........+.+.........+............+.........+.....................+..+.......+......+..+.+...............+..+............+...+.......+..+.............+......+..+............+.+............+...+......+.....................+.....+.......+........+....+......+.....+...+......+......+...................+...........+...+......+.........+.........+...+....+....................+....+.....+.....................+..........+......+..+...+...+.......+..+.+.....+.+..............+.......+.....+....+...+.................+.+..............+...+.+..+....+...........+.........+...+...............+.+..+....+...+...+........+....+...+........+....+.................+..........+...+......+......+...+.....+......+.........+......+......+...............+....+..............+............+.........+..........+..+++++ [e2e-predictor] ..+...........+.....................+...+....+......+.....+++++++++++++++++++++++++++++++++++++++++++++*...+......+.+..+......+.+...+..+.+........+.+.....+.............+++++++++++++++++++++++++++++++++++++++++++++*...........+.....+.............+.....+.......+......+.........+............+...+.....+...+...........................+.......+...+..+......+...+....+..+.+.................+....................................+....+..+...................+...........+.+..+......+............+.........+..................+................+...........+...+............+.........+.+...+...+.....+...+......+......+...+................+...+.....+..........+...+........+.......+...........................+..+.............+...............+..+...+.+......+...............+........+.+..+.......+...+..+.+...............+..............+..........+..+...............+......+.........+...+.+..+....+........+............+..........+......+...+...........+......+...+.+......+........+....+++++ [e2e-predictor] ----- [e2e-predictor] Certificate request self-signature ok [e2e-predictor] subject=CN=custom, O=Red Hat [e2e-predictor] secret/seaweedfs-tls-custom created [e2e-predictor] deployment.apps/seaweedfs-tls-custom patched [e2e-predictor] Waiting for patched seaweedfs-tls-custom deployment to be ready... [e2e-predictor] Waiting for deployment "seaweedfs-tls-custom" rollout to finish: 0 out of 1 new replicas have been updated... [e2e-predictor] Waiting for deployment "seaweedfs-tls-custom" rollout to finish: 0 out of 1 new replicas have been updated... [e2e-predictor] Waiting for deployment "seaweedfs-tls-custom" rollout to finish: 0 out of 1 new replicas have been updated... [e2e-predictor] Waiting for deployment "seaweedfs-tls-custom" rollout to finish: 0 of 1 updated replicas are available... [e2e-predictor] deployment "seaweedfs-tls-custom" successfully rolled out [e2e-predictor] job.batch/s3-tls-init-custom created [e2e-predictor] Waiting for S3 TLS init job to complete... [e2e-predictor] job.batch/s3-tls-init-custom condition met [e2e-predictor] Namespace kserve exists. [e2e-predictor] Cleaning up existing serving TLS SeaweedFS resources for idempotency... [e2e-predictor] secret/seaweedfs-tls-serving-artifact serverside-applied [e2e-predictor] service/seaweedfs-tls-serving-service serverside-applied [e2e-predictor] deployment.apps/seaweedfs-tls-serving serverside-applied [e2e-predictor] Waiting for seaweedfs-tls-serving deployment to be ready... [e2e-predictor] Waiting for deployment "seaweedfs-tls-serving" rollout to finish: 0 of 1 updated replicas are available... [e2e-predictor] deployment "seaweedfs-tls-serving" successfully rolled out [e2e-predictor] Configuring SeaweedFS S3 for TLS with Openshift serving certificate and adding models to storage [e2e-predictor] job.batch/s3-tls-init-serving created [e2e-predictor] Waiting for S3 TLS init job to complete... [e2e-predictor] job.batch/s3-tls-init-serving condition met [e2e-predictor] Prepare CI namespace and install ServingRuntimes [e2e-predictor] Setting up CI namespace: kserve-ci-e2e-test [e2e-predictor] Tearing down CI namespace: kserve-ci-e2e-test [e2e-predictor] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-predictor] CI namespace teardown complete [e2e-predictor] Creating namespace kserve-ci-e2e-test [e2e-predictor] namespace/kserve-ci-e2e-test created [e2e-predictor] Applying S3 artifact secret [e2e-predictor] secret/mlpipeline-s3-artifact created [e2e-predictor] Applying storage-config secret [e2e-predictor] secret/storage-config created [e2e-predictor] Applying SeaweedFS S3 credentials secret [e2e-predictor] secret/seaweedfs-s3-creds created [e2e-predictor] Linking seaweedfs-s3-creds to default service account [e2e-predictor] Creating odh-trusted-ca-bundle configmap [e2e-predictor] configmap/odh-trusted-ca-bundle created [e2e-predictor] Installing ServingRuntimes [e2e-predictor] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-predictor] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-predictor] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-predictor] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-predictor] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-predictor] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-predictor] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-predictor] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-predictor] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-predictor] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-predictor] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-predictor] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-predictor] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-predictor] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-predictor] CI namespace setup complete [e2e-predictor] Setup complete [e2e-predictor] === E2E cluster / operator summary === [e2e-predictor] Client Version: 4.20.11 [e2e-predictor] Kustomize Version: v5.6.0 [e2e-predictor] Server Version: 4.21.26 [e2e-predictor] Kubernetes Version: v1.34.9 [e2e-predictor] ClusterVersion desired: 4.21.26 [e2e-predictor] ClusterVersion history (latest): 4.21.26 (Completed) [e2e-predictor] CSVs in openshift-keda: [e2e-predictor] custom-metrics-autoscaler.v2.19.0-2 Succeeded [e2e-predictor] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-predictor] KServe deployments in kserve: [e2e-predictor] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:a627ec30debc454af106c2c34af695e96eccb25d564ea70da2ca083e0bc41d91 [e2e-predictor] imageID: quay.io/opendatahub/kserve-controller@sha256:406c6b521a73de6d8758208f8b01fbda1aca8cbb849ba76749aa41667d674b0d [e2e-predictor] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:odh-incubating [e2e-predictor] imageID: quay.io/opendatahub/odh-model-controller@sha256:b5c835d640b650f164ba7f36d7008760c88e2974efa65853ee167323c441580c [e2e-predictor] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller:odh-master [e2e-predictor] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:4979024cc63df7670b194e7a1cd0e5da6ae8163e531a0d64f685b407268c2f39 [e2e-predictor] === End E2E cluster / operator summary === [e2e-predictor] /workspace/source [e2e-predictor] CA certificate extracted [e2e-predictor] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-predictor] Run E2E tests: predictor or kserve_on_openshift [e2e-predictor] Starting E2E functional tests ... [e2e-predictor] Parallelism requested for pytest is 1 [e2e-predictor] ============================= test session starts ============================== [e2e-predictor] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-predictor] cachedir: .pytest_cache [e2e-predictor] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.124.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'asyncio': '0.23.8', 'xdist': '3.6.1', 'json-report': '1.5.0', 'httpx': '0.30.0', 'cov': '5.0.0', 'metadata': '3.1.1', 'anyio': '4.9.0'}, 'PLATFORM': 'el9'} [e2e-predictor] rootdir: /workspace/source/test/e2e [e2e-predictor] configfile: pytest.ini [e2e-predictor] plugins: asyncio-0.23.8, xdist-3.6.1, json-report-1.5.0, httpx-0.30.0, cov-5.0.0, metadata-3.1.1, anyio-4.9.0 [e2e-predictor] asyncio: mode=Mode.STRICT [e2e-predictor] created: 1/1 worker [e2e-predictor] 1 worker [65 items] [e2e-predictor] [e2e-predictor] scheduling tests via WorkStealingScheduling [e2e-predictor] [e2e-predictor] batcher/test_batcher.py::test_batcher 2026-07-28 16:18:05.619 4494 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-predictor] 2026-07-28 16:19:45.929 4494 kserve.trace Host from isvc status URL = isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:19:45.929 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:19:45.929 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:19:45.929 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:19:45.929 4494 kserve.trace Using external route host: isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:19:45.929 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:19:45.931 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:19:45.931 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:19:45.931 4494 kserve.trace base url = https://isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:19:45.931 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:19:46.038 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:19:46.038 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:19:46.038 4494 kserve.trace base url = https://isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:19:46.038 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:19:46.039 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:19:46.039 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:19:46.039 4494 kserve.trace base url = https://isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:19:46.039 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:19:46.039 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:19:46.039 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:19:46.039 4494 kserve.trace base url = https://isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:19:46.039 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-batcher-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:19:51.073 4494 kserve.trace Got response [{'message': '', 'batchId': '90c5c3e0-d7f7-4f3d-95d5-fa85127adaf0', 'predictions': [1, 1]}, {'message': '', 'batchId': '90c5c3e0-d7f7-4f3d-95d5-fa85127adaf0', 'predictions': [1, 1]}, {'message': '', 'batchId': '90c5c3e0-d7f7-4f3d-95d5-fa85127adaf0', 'predictions': [1, 1]}, {'message': '', 'batchId': '90c5c3e0-d7f7-4f3d-95d5-fa85127adaf0', 'predictions': [1, 1]}] [e2e-predictor] 2026-07-28 16:19:51.073 4494 kserve.trace INFO [utils.py:predict():148] Got response [{'message': '', 'batchId': '90c5c3e0-d7f7-4f3d-95d5-fa85127adaf0', 'predictions': [1, 1]}, {'message': '', 'batchId': '90c5c3e0-d7f7-4f3d-95d5-fa85127adaf0', 'predictions': [1, 1]}, {'message': '', 'batchId': '90c5c3e0-d7f7-4f3d-95d5-fa85127adaf0', 'predictions': [1, 1]}, {'message': '', 'batchId': '90c5c3e0-d7f7-4f3d-95d5-fa85127adaf0', 'predictions': [1, 1]}] [e2e-predictor] [e2e-predictor] [gw0] PASSED batcher/test_batcher.py::test_batcher [e2e-predictor] batcher/test_batcher_custom_port.py::test_batcher_custom_port 2026-07-28 16:21:21.299 4494 kserve.trace Host from isvc status URL = isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:21:21.299 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:21:21.299 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:21:21.299 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:21:21.299 4494 kserve.trace Using external route host: isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:21:21.299 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:21:21.301 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:21:21.301 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:21:21.301 4494 kserve.trace base url = https://isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:21:21.301 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:21:21.301 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:21:21.301 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:21:21.302 4494 kserve.trace base url = https://isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:21:21.302 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:21:21.302 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:21:21.302 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:21:21.302 4494 kserve.trace base url = https://isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:21:21.302 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:21:21.303 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:21:21.303 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:21:21.303 4494 kserve.trace base url = https://isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:21:21.303 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-batcher-custom-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:21:26.337 4494 kserve.trace Got response [{'message': '', 'batchId': '60a83b0d-4397-43a6-95cb-165ac7a4946a', 'predictions': [1, 1]}, {'message': '', 'batchId': '60a83b0d-4397-43a6-95cb-165ac7a4946a', 'predictions': [1, 1]}, {'message': '', 'batchId': '60a83b0d-4397-43a6-95cb-165ac7a4946a', 'predictions': [1, 1]}, {'message': '', 'batchId': '60a83b0d-4397-43a6-95cb-165ac7a4946a', 'predictions': [1, 1]}] [e2e-predictor] 2026-07-28 16:21:26.337 4494 kserve.trace INFO [utils.py:predict():148] Got response [{'message': '', 'batchId': '60a83b0d-4397-43a6-95cb-165ac7a4946a', 'predictions': [1, 1]}, {'message': '', 'batchId': '60a83b0d-4397-43a6-95cb-165ac7a4946a', 'predictions': [1, 1]}, {'message': '', 'batchId': '60a83b0d-4397-43a6-95cb-165ac7a4946a', 'predictions': [1, 1]}, {'message': '', 'batchId': '60a83b0d-4397-43a6-95cb-165ac7a4946a', 'predictions': [1, 1]}] [e2e-predictor] [e2e-predictor] [gw0] PASSED batcher/test_batcher_custom_port.py::test_batcher_custom_port [e2e-predictor] custom/test_custom_model_grpc.py::test_custom_model_grpc [e2e-predictor] [gw0] SKIPPED custom/test_custom_model_grpc.py::test_custom_model_grpc [e2e-predictor] custom/test_ray.py::test_custom_model_http_ray [e2e-predictor] [gw0] SKIPPED custom/test_ray.py::test_custom_model_http_ray [e2e-predictor] logger/test_logger.py::test_kserve_logger 2026-07-28 16:23:16.565 4494 kserve.trace Host from isvc status URL = isvc-logger-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:23:16.565 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-logger-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:23:16.565 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:23:16.565 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:23:16.565 4494 kserve.trace Using external route host: isvc-logger-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:23:16.565 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-logger-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:23:16.565 4494 kserve.trace Sending Header = {'Host': 'isvc-logger-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:23:16.565 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-logger-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:23:16.566 4494 kserve.trace base url = https://isvc-logger-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:23:16.566 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-logger-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:23:16.597 4494 kserve.trace Got response {'predictions': [1, 1]} [e2e-predictor] 2026-07-28 16:23:16.597 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [1, 1]} [e2e-predictor] [e2e-predictor] [gw0] PASSED logger/test_logger.py::test_kserve_logger [e2e-predictor] predictor/test_autogluon.py::test_autogluon_runtime_kserve_v1 2026-07-28 16:28:20.414 4494 kserve.trace Host from isvc status URL = isvc-autogluon-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:28:20.414 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-autogluon-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:28:20.414 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:28:20.414 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:28:20.414 4494 kserve.trace Using external route host: isvc-autogluon-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:28:20.414 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-autogluon-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:28:20.414 4494 kserve.trace Sending Header = {'Host': 'isvc-autogluon-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:28:20.414 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-autogluon-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:28:20.414 4494 kserve.trace base url = https://isvc-autogluon-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:28:20.414 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-autogluon-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:28:26.892 4494 kserve.trace Got response {'predictions': [0, 1]} [e2e-predictor] 2026-07-28 16:28:26.892 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [0, 1]} [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_autogluon.py::test_autogluon_runtime_kserve_v1 [e2e-predictor] predictor/test_autogluon.py::test_autogluon_runtime_kserve_v2 2026-07-28 16:30:42.227 4494 kserve.trace Host from isvc status URL = isvc-autogluon-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:30:42.227 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-autogluon-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:30:42.227 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:30:42.227 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:30:42.227 4494 kserve.trace Using external route host: isvc-autogluon-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:30:42.227 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-autogluon-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:30:42.227 4494 kserve.trace Sending Header = {'Host': 'isvc-autogluon-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:30:42.227 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-autogluon-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:30:42.227 4494 kserve.trace base url = https://isvc-autogluon-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:30:42.227 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-autogluon-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:30:48.561 4494 kserve.trace Got response "id": "126ae7f3-90d3-4af9-9216-8158c2245aa9","model_name": "isvc-autogluon-v2","outputs": ["name": "predictions","shape": [2],"datatype": "INT64","data": [0, 1],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] 2026-07-28 16:30:48.561 4494 kserve.trace INFO [utils.py:predict():148] Got response "id": "126ae7f3-90d3-4af9-9216-8158c2245aa9","model_name": "isvc-autogluon-v2","outputs": ["name": "predictions","shape": [2],"datatype": "INT64","data": [0, 1],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_autogluon.py::test_autogluon_runtime_kserve_v2 [e2e-predictor] predictor/test_autogluon.py::test_autogluon_runtime_kserve_v2_input_variants 2026-07-28 16:33:03.875 4494 kserve.trace Host from isvc status URL = isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:03.875 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:03.876 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:33:03.876 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:33:03.876 4494 kserve.trace Using external route host: isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:03.876 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:03.876 4494 kserve.trace Sending Header = {'Host': 'isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:33:03.876 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:33:03.876 4494 kserve.trace base url = https://isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:03.876 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:09.836 4494 kserve.trace Got response "id": "c84e3be1-e56e-4660-8353-cdc4081751f5","model_name": "isvc-autogluon-v2-variants","outputs": ["name": "predictions","shape": [2],"datatype": "INT64","data": [0, 1],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] 2026-07-28 16:33:09.836 4494 kserve.trace INFO [utils.py:predict():148] Got response "id": "c84e3be1-e56e-4660-8353-cdc4081751f5","model_name": "isvc-autogluon-v2-variants","outputs": ["name": "predictions","shape": [2],"datatype": "INT64","data": [0, 1],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] 2026-07-28 16:33:09.876 4494 kserve.trace Host from isvc status URL = isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:09.876 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:09.876 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:33:09.876 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:33:09.876 4494 kserve.trace Using external route host: isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:09.876 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:09.876 4494 kserve.trace Sending Header = {'Host': 'isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:33:09.876 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:33:09.876 4494 kserve.trace base url = https://isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:09.876 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:10.056 4494 kserve.trace Got response "id": "12727a79-a352-4472-82b3-e731275f0487","model_name": "isvc-autogluon-v2-variants","outputs": ["name": "predictions","shape": [2],"datatype": "INT64","data": [0, 1],"parameters": {}],"parameters": None,"from_grpc": False [e2e-predictor] 2026-07-28 16:33:10.056 4494 kserve.trace INFO [utils.py:predict():148] Got response "id": "12727a79-a352-4472-82b3-e731275f0487","model_name": "isvc-autogluon-v2-variants","outputs": ["name": "predictions","shape": [2],"datatype": "INT64","data": [0, 1],"parameters": {}],"parameters": None,"from_grpc": False [e2e-predictor] 2026-07-28 16:33:10.097 4494 kserve.trace Host from isvc status URL = isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:10.097 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:10.097 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:33:10.097 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:33:10.097 4494 kserve.trace Using external route host: isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:10.097 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:10.098 4494 kserve.trace Sending Header = {'Host': 'isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:33:10.098 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:33:10.098 4494 kserve.trace base url = https://isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:10.098 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-autogluon-v2-variants-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:33:10.282 4494 kserve.trace Got response "id": "1748a74c-ab30-4824-9eb5-ef0496825fa6","model_name": "isvc-autogluon-v2-variants","outputs": ["name": "predictions","shape": [2],"datatype": "INT64","data": [0, 1],"parameters": {}],"parameters": None,"from_grpc": False [e2e-predictor] 2026-07-28 16:33:10.282 4494 kserve.trace INFO [utils.py:predict():148] Got response "id": "1748a74c-ab30-4824-9eb5-ef0496825fa6","model_name": "isvc-autogluon-v2-variants","outputs": ["name": "predictions","shape": [2],"datatype": "INT64","data": [0, 1],"parameters": {}],"parameters": None,"from_grpc": False [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_autogluon.py::test_autogluon_runtime_kserve_v2_input_variants [e2e-predictor] predictor/test_autogluon.py::test_autogluon_runtime_kserve_v2_storage_uri_without_trailing_slash 2026-07-28 16:35:05.547 4494 kserve.trace Host from isvc status URL = isvc-autogluon-v2-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:35:05.547 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-autogluon-v2-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:35:05.547 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:35:05.547 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:35:05.547 4494 kserve.trace Using external route host: isvc-autogluon-v2-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:35:05.547 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-autogluon-v2-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:35:05.547 4494 kserve.trace Sending Header = {'Host': 'isvc-autogluon-v2-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:35:05.547 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-autogluon-v2-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:35:05.547 4494 kserve.trace base url = https://isvc-autogluon-v2-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:35:05.547 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-autogluon-v2-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:35:11.764 4494 kserve.trace Got response "id": "f950ff5d-4055-4382-86b9-fdc149eddb00","model_name": "isvc-autogluon-v2-noslash","outputs": ["name": "predictions","shape": [2],"datatype": "INT64","data": [0, 1],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] 2026-07-28 16:35:11.764 4494 kserve.trace INFO [utils.py:predict():148] Got response "id": "f950ff5d-4055-4382-86b9-fdc149eddb00","model_name": "isvc-autogluon-v2-noslash","outputs": ["name": "predictions","shape": [2],"datatype": "INT64","data": [0, 1],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_autogluon.py::test_autogluon_runtime_kserve_v2_storage_uri_without_trailing_slash [e2e-predictor] predictor/test_autogluon_timeseries.py::test_autogluon_timeseries_runtime_kserve_v1 2026-07-28 16:35:46.983 4494 kserve.trace Host from isvc status URL = isvc-autogluon-ts-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:35:46.983 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-autogluon-ts-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:35:46.984 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:35:46.984 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:35:46.984 4494 kserve.trace Using external route host: isvc-autogluon-ts-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:35:46.984 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-autogluon-ts-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:35:46.984 4494 kserve.trace Sending Header = {'Host': 'isvc-autogluon-ts-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:35:46.984 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-autogluon-ts-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:35:46.984 4494 kserve.trace base url = https://isvc-autogluon-ts-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:35:46.984 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-autogluon-ts-v1-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:35:59.621 4494 kserve.trace Got response {'predictions': [{'item_id': '0', 'timestamp': '2020-01-03T01:00:00', 'mean': 101.86467742919922, '0.1': 101.45105843997753, '0.2': 101.59304534407902, '0.3': 101.69542789149415, '0.4': 101.78291000326132, '0.5': 101.86467742919922, '0.6': 101.94644485513712, '0.7': 102.03392696690429, '0.8': 102.13630951431942, '0.9': 102.27829641842091}, {'item_id': '0', 'timestamp': '2020-01-03T02:00:00', 'mean': 102.43192291259766, '0.1': 101.8469773283853, '0.2': 102.04777713384499, '0.3': 102.19256792094977, '0.4': 102.31628630987593, '0.5': 102.43192291259766, '0.6': 102.54755951531938, '0.7': 102.67127790424554, '0.8': 102.81606869135032, '0.9': 103.01686849681002}, {'item_id': '0', 'timestamp': '2020-01-03T03:00:00', 'mean': 102.62620544433594, '0.1': 101.90979634002869, '0.2': 102.15572487194187, '0.3': 102.33305664587321, '0.4': 102.48458010820737, '0.5': 102.62620544433594, '0.6': 102.76783078046451, '0.7': 102.91935424279866, '0.8': 103.09668601673, '0.9': 103.34261454864318}]} [e2e-predictor] 2026-07-28 16:35:59.621 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [{'item_id': '0', 'timestamp': '2020-01-03T01:00:00', 'mean': 101.86467742919922, '0.1': 101.45105843997753, '0.2': 101.59304534407902, '0.3': 101.69542789149415, '0.4': 101.78291000326132, '0.5': 101.86467742919922, '0.6': 101.94644485513712, '0.7': 102.03392696690429, '0.8': 102.13630951431942, '0.9': 102.27829641842091}, {'item_id': '0', 'timestamp': '2020-01-03T02:00:00', 'mean': 102.43192291259766, '0.1': 101.8469773283853, '0.2': 102.04777713384499, '0.3': 102.19256792094977, '0.4': 102.31628630987593, '0.5': 102.43192291259766, '0.6': 102.54755951531938, '0.7': 102.67127790424554, '0.8': 102.81606869135032, '0.9': 103.01686849681002}, {'item_id': '0', 'timestamp': '2020-01-03T03:00:00', 'mean': 102.62620544433594, '0.1': 101.90979634002869, '0.2': 102.15572487194187, '0.3': 102.33305664587321, '0.4': 102.48458010820737, '0.5': 102.62620544433594, '0.6': 102.76783078046451, '0.7': 102.91935424279866, '0.8': 103.09668601673, '0.9': 103.34261454864318}]} [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_autogluon_timeseries.py::test_autogluon_timeseries_runtime_kserve_v1 [e2e-predictor] predictor/test_autogluon_timeseries.py::test_autogluon_timeseries_runtime_kserve_v1_storage_uri_without_trailing_slash 2026-07-28 16:38:44.458 4494 kserve.trace Host from isvc status URL = isvc-autogluon-ts-v1-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:38:44.458 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-autogluon-ts-v1-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:38:44.458 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:38:44.458 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:38:44.458 4494 kserve.trace Using external route host: isvc-autogluon-ts-v1-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:38:44.458 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-autogluon-ts-v1-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:38:44.459 4494 kserve.trace Sending Header = {'Host': 'isvc-autogluon-ts-v1-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:38:44.459 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-autogluon-ts-v1-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:38:44.459 4494 kserve.trace base url = https://isvc-autogluon-ts-v1-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:38:44.459 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-autogluon-ts-v1-noslash-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:38:49.564 4494 kserve.trace Got response {'predictions': [{'item_id': 'H200', 'timestamp': '1750-02-10T06:00:00', 'mean': 17.500171661376953, '0.1': 17.049267429998, '0.2': 17.20405359196854, '0.3': 17.31566530480529, '0.4': 17.411033398131973, '0.5': 17.500171661376953, '0.6': 17.589309924621933, '0.7': 17.684678017948617, '0.8': 17.796289730785368, '0.9': 17.951075892755906}, {'item_id': 'H200', 'timestamp': '1750-02-10T07:00:00', 'mean': 17.200218200683594, '0.1': 16.562543321336065, '0.2': 16.781444010862476, '0.3': 16.9392868088759, '0.4': 17.07415765987616, '0.5': 17.200218200683594, '0.6': 17.326278741491027, '0.7': 17.46114959249129, '0.8': 17.61899239050471, '0.9': 17.837893080031122}, {'item_id': 'H200', 'timestamp': '1750-02-10T08:00:00', 'mean': 16.800273895263672, '0.1': 16.019284857167534, '0.2': 16.28738235400909, '0.3': 16.48069951136213, '0.4': 16.645881894424917, '0.5': 16.800273895263672, '0.6': 16.954665896102426, '0.7': 17.119848279165215, '0.8': 17.313165436518254, '0.9': 17.58126293335981}]} [e2e-predictor] 2026-07-28 16:38:49.564 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [{'item_id': 'H200', 'timestamp': '1750-02-10T06:00:00', 'mean': 17.500171661376953, '0.1': 17.049267429998, '0.2': 17.20405359196854, '0.3': 17.31566530480529, '0.4': 17.411033398131973, '0.5': 17.500171661376953, '0.6': 17.589309924621933, '0.7': 17.684678017948617, '0.8': 17.796289730785368, '0.9': 17.951075892755906}, {'item_id': 'H200', 'timestamp': '1750-02-10T07:00:00', 'mean': 17.200218200683594, '0.1': 16.562543321336065, '0.2': 16.781444010862476, '0.3': 16.9392868088759, '0.4': 17.07415765987616, '0.5': 17.200218200683594, '0.6': 17.326278741491027, '0.7': 17.46114959249129, '0.8': 17.61899239050471, '0.9': 17.837893080031122}, {'item_id': 'H200', 'timestamp': '1750-02-10T08:00:00', 'mean': 16.800273895263672, '0.1': 16.019284857167534, '0.2': 16.28738235400909, '0.3': 16.48069951136213, '0.4': 16.645881894424917, '0.5': 16.800273895263672, '0.6': 16.954665896102426, '0.7': 17.119848279165215, '0.8': 17.313165436518254, '0.9': 17.58126293335981}]} [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_autogluon_timeseries.py::test_autogluon_timeseries_runtime_kserve_v1_storage_uri_without_trailing_slash [e2e-predictor] predictor/test_autoscaling.py::test_sklearn_kserve_concurrency [e2e-predictor] [gw0] SKIPPED predictor/test_autoscaling.py::test_sklearn_kserve_concurrency [e2e-predictor] predictor/test_autoscaling.py::test_sklearn_kserve_rps [e2e-predictor] [gw0] SKIPPED predictor/test_autoscaling.py::test_sklearn_kserve_rps [e2e-predictor] predictor/test_canary.py::test_canary_rollout [e2e-predictor] [gw0] SKIPPED predictor/test_canary.py::test_canary_rollout [e2e-predictor] predictor/test_canary.py::test_canary_rollout_runtime [e2e-predictor] [gw0] SKIPPED predictor/test_canary.py::test_canary_rollout_runtime [e2e-predictor] predictor/test_canary_raw_deployment.py::test_canary_create [e2e-predictor] [gw0] PASSED predictor/test_canary_raw_deployment.py::test_canary_create [e2e-predictor] predictor/test_canary_raw_deployment.py::test_canary_promote [e2e-predictor] [gw0] PASSED predictor/test_canary_raw_deployment.py::test_canary_promote [e2e-predictor] predictor/test_canary_raw_deployment.py::test_canary_rollback [e2e-predictor] [gw0] PASSED predictor/test_canary_raw_deployment.py::test_canary_rollback [e2e-predictor] predictor/test_canary_raw_deployment.py::test_canary_force_stop [e2e-predictor] [gw0] PASSED predictor/test_canary_raw_deployment.py::test_canary_force_stop [e2e-predictor] predictor/test_lightgbm.py::test_lightgbm_kserve 2026-07-28 16:54:41.326 4494 kserve.trace Host from isvc status URL = isvc-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:54:41.326 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:54:41.326 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:54:41.326 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:54:41.326 4494 kserve.trace Using external route host: isvc-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:54:41.326 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:54:41.327 4494 kserve.trace Sending Header = {'Host': 'isvc-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:54:41.327 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:54:41.327 4494 kserve.trace base url = https://isvc-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:54:41.327 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:54:41.366 4494 kserve.trace Got response {'predictions': [[0.9999378629898321, 4.415799218835629e-05, 1.797901797954119e-05]]} [e2e-predictor] 2026-07-28 16:54:41.366 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [[0.9999378629898321, 4.415799218835629e-05, 1.797901797954119e-05]]} [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_lightgbm.py::test_lightgbm_kserve [e2e-predictor] predictor/test_lightgbm.py::test_lightgbm_runtime_kserve 2026-07-28 16:56:26.644 4494 kserve.trace Host from isvc status URL = isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.644 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.645 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:56:26.645 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:56:26.645 4494 kserve.trace Using external route host: isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.645 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.645 4494 kserve.trace Sending Header = {'Host': 'isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:56:26.645 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:56:26.645 4494 kserve.trace base url = https://isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.645 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.677 4494 kserve.trace Got response {'predictions': [[0.9999378629898321, 4.415799218835629e-05, 1.797901797954119e-05]]} [e2e-predictor] 2026-07-28 16:56:26.677 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [[0.9999378629898321, 4.415799218835629e-05, 1.797901797954119e-05]]} [e2e-predictor] 2026-07-28 16:56:26.718 4494 kserve.trace Host from isvc status URL = isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.718 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.719 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:56:26.719 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:56:26.719 4494 kserve.trace Using external route host: isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.719 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.751 4494 kserve.trace Sending Header = {'Host': 'isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:56:26.751 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:56:26.752 4494 kserve.trace base url = https://isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.752 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.781 4494 kserve.trace Got response {'predictions': [[0.9971662536216064, 0.0018206154870803392, 0.0010131308913132729], [0.9999572529695757, 2.544262359421485e-05, 1.7304406830185675e-05], [0.9983992430047839, 0.0008007832660759804, 0.0007999737291403532], [0.9983992430047839, 0.0008007832660759804, 0.0007999737291403532]]} [e2e-predictor] 2026-07-28 16:56:26.781 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [[0.9971662536216064, 0.0018206154870803392, 0.0010131308913132729], [0.9999572529695757, 2.544262359421485e-05, 1.7304406830185675e-05], [0.9983992430047839, 0.0008007832660759804, 0.0007999737291403532], [0.9983992430047839, 0.0008007832660759804, 0.0007999737291403532]]} [e2e-predictor] 2026-07-28 16:56:26.818 4494 kserve.trace Host from isvc status URL = isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.818 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.818 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 16:56:26.818 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 16:56:26.818 4494 kserve.trace Using external route host: isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.818 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.828 4494 kserve.trace Sending Header = {'Host': 'isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:56:26.828 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 16:56:26.828 4494 kserve.trace base url = https://isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.828 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-lightgbm-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 16:56:26.981 4494 kserve.trace Got response {'predictions': [[0.9971662536216064, 0.0018206154870803392, 0.0010131308913132729], [0.9999572529695757, 2.544262359421485e-05, 1.7304406830185675e-05], [0.9983992430047839, 0.0008007832660759804, 0.0007999737291403532]]} [e2e-predictor] 2026-07-28 16:56:26.981 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [[0.9971662536216064, 0.0018206154870803392, 0.0010131308913132729], [0.9999572529695757, 2.544262359421485e-05, 1.7304406830185675e-05], [0.9983992430047839, 0.0008007832660759804, 0.0007999737291403532]]} [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_lightgbm.py::test_lightgbm_runtime_kserve [e2e-predictor] predictor/test_lightgbm.py::test_lightgbm_v2_runtime_mlserver 2026-07-28 17:04:04.450 4494 kserve.trace Host from isvc status URL = isvc-lightgbm-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:04:04.450 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-lightgbm-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:04:04.451 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:04:04.451 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:04:04.451 4494 kserve.trace Using external route host: isvc-lightgbm-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:04:04.451 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-lightgbm-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:04:04.451 4494 kserve.trace Sending Header = {'Host': 'isvc-lightgbm-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:04:04.451 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-lightgbm-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:04:04.451 4494 kserve.trace base url = https://isvc-lightgbm-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:04:04.451 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-lightgbm-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:04:04.497 4494 kserve.trace Got response "id": "9840199b-7c58-4c21-b264-259df2a067ca","model_name": "isvc-lightgbm-v2-runtime","outputs": ["name": "predict","shape": [2, 3],"datatype": "FP64","data": [8.796664107010673e-06, 0.9992300031041593, 0.0007612002317336916, 4.974786820804187e-06, 0.9999919650711493, 3.0601420299625077e-06],"parameters": {'content_type': 'np'}],"parameters": {},"from_grpc": False [e2e-predictor] 2026-07-28 17:04:04.497 4494 kserve.trace INFO [utils.py:predict():148] Got response "id": "9840199b-7c58-4c21-b264-259df2a067ca","model_name": "isvc-lightgbm-v2-runtime","outputs": ["name": "predict","shape": [2, 3],"datatype": "FP64","data": [8.796664107010673e-06, 0.9992300031041593, 0.0007612002317336916, 4.974786820804187e-06, 0.9999919650711493, 3.0601420299625077e-06],"parameters": {'content_type': 'np'}],"parameters": {},"from_grpc": False [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_lightgbm.py::test_lightgbm_v2_runtime_mlserver [e2e-predictor] predictor/test_lightgbm.py::test_lightgbm_v2_kserve 2026-07-28 17:04:29.731 4494 kserve.trace Host from isvc status URL = isvc-lightgbm-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:04:29.731 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-lightgbm-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:04:29.731 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:04:29.731 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:04:29.731 4494 kserve.trace Using external route host: isvc-lightgbm-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:04:29.731 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-lightgbm-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:04:29.732 4494 kserve.trace Sending Header = {'Host': 'isvc-lightgbm-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:04:29.732 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-lightgbm-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:04:29.732 4494 kserve.trace base url = https://isvc-lightgbm-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:04:29.732 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-lightgbm-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:04:29.761 4494 kserve.trace Got response "id": "843273f9-08f6-410c-acb9-27e91623db97","model_name": "isvc-lightgbm-v2-kserve","outputs": ["name": "output-0","shape": [2, 3],"datatype": "FP64","data": [8.796664107010673e-06, 0.9992300031041593, 0.0007612002317336916, 4.974786820804187e-06, 0.9999919650711493, 3.0601420299625077e-06],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] 2026-07-28 17:04:29.761 4494 kserve.trace INFO [utils.py:predict():148] Got response "id": "843273f9-08f6-410c-acb9-27e91623db97","model_name": "isvc-lightgbm-v2-kserve","outputs": ["name": "output-0","shape": [2, 3],"datatype": "FP64","data": [8.796664107010673e-06, 0.9992300031041593, 0.0007612002317336916, 4.974786820804187e-06, 0.9999919650711493, 3.0601420299625077e-06],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_lightgbm.py::test_lightgbm_v2_kserve [e2e-predictor] predictor/test_lightgbm.py::test_lightgbm_v2_grpc [e2e-predictor] [gw0] SKIPPED predictor/test_lightgbm.py::test_lightgbm_v2_grpc [e2e-predictor] predictor/test_mlflow.py::test_mlflow_v2_runtime_kserve 2026-07-28 17:07:33.504 4494 kserve.trace Host from isvc status URL = isvc-mlflow-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:07:33.504 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-mlflow-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:07:33.505 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:07:33.505 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:07:33.505 4494 kserve.trace Using external route host: isvc-mlflow-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:07:33.505 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-mlflow-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:07:33.505 4494 kserve.trace Sending Header = {'Host': 'isvc-mlflow-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:07:33.505 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-mlflow-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:07:33.506 4494 kserve.trace base url = https://isvc-mlflow-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:07:33.506 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-mlflow-v2-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:07:33.540 4494 kserve.trace Got response "id": "693493fd-8686-476e-8204-34664e982ab0","model_name": "isvc-mlflow-v2-runtime","outputs": ["name": "output-1","shape": [1, 1],"datatype": "FP64","data": [5.576883936610762],"parameters": {'content_type': 'np'}],"parameters": {'content_type': 'np'},"from_grpc": False [e2e-predictor] 2026-07-28 17:07:33.540 4494 kserve.trace INFO [utils.py:predict():148] Got response "id": "693493fd-8686-476e-8204-34664e982ab0","model_name": "isvc-mlflow-v2-runtime","outputs": ["name": "output-1","shape": [1, 1],"datatype": "FP64","data": [5.576883936610762],"parameters": {'content_type': 'np'}],"parameters": {'content_type': 'np'},"from_grpc": False [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_mlflow.py::test_mlflow_v2_runtime_kserve [e2e-predictor] predictor/test_multi_container_probing.py::test_multi_container_probing 2026-07-28T17:13:21.972207+00:00 timeout_sampler INFO Waiting for 60 seconds [0:01:00], retry every 2 seconds. (Function: e2e.predictor.test_multi_container_probing..lambda: k8s_client.service_name.get_deployment) [e2e-predictor] 2026-07-28T17:13:22.011195+00:00 timeout_sampler INFO Elapsed time: 0.0003085136413574219 [0:00:00.000309] [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_multi_container_probing.py::test_multi_container_probing [e2e-predictor] predictor/test_paddle.py::test_paddle 2026-07-28 17:18:56.057 4494 kserve.trace Host from isvc status URL = isvc-paddle-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:18:56.057 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-paddle-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:18:56.057 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:18:56.057 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:18:56.057 4494 kserve.trace Using external route host: isvc-paddle-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:18:56.057 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-paddle-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:18:56.118 4494 kserve.trace Sending Header = {'Host': 'isvc-paddle-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:18:56.118 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-paddle-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:18:56.119 4494 kserve.trace base url = https://isvc-paddle-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:18:56.119 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-paddle-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:18:57.737 4494 kserve.trace Got response {'predictions': [[6.736569968524009e-09, 1.153582651625129e-08, 5.142186410012073e-08, 6.647101002954514e-08, 4.094442118685038e-08, 1.3402272713847196e-07, 9.35544619551365e-08, 2.8935589924117266e-08, 6.845294819868286e-08, 7.680520752728626e-08, 2.0334418877610005e-06, 1.1085519417974865e-06, 2.3477346644540376e-07, 6.581955744877632e-07, 0.00012372950732242316, 4.2878312456196e-07, 6.419879809982376e-06, 0.9993487000465393, 7.371911488007754e-05, 3.101079710177146e-05, 5.602713827101979e-06, 2.1862256289750803e-06, 1.9543842100233633e-08, 3.728848128048412e-07, 4.29031416615544e-07, 1.8250955235998845e-07, 7.159837878134567e-08, 9.231503561579757e-09, 6.469168170042394e-07, 7.031603299623157e-09, 4.4511686780879245e-08, 1.2455829789814743e-07, 9.446210924579646e-08, 4.3476589439706004e-08, 4.6581672563661414e-07, 6.797636586952649e-08, 2.1060014887552825e-07, 2.2604888627597575e-08, 1.4311326879123953e-07, 7.951216218771151e-08, 1.2341666888460168e-07, 1.0921577313638409e-06, 1.5243762391037308e-05, 3.1172930903267115e-07, 2.415180802017858e-07, 6.863702850523623e-08, 8.46760883632669e-08, 9.424560687421035e-08, 1.0219084245477461e-08, 3.3770398033539095e-08, 3.6928447855189006e-08, 1.3693848188722768e-07, 1.067415098532365e-07, 2.5994486918534676e-07, 3.486601087843155e-07, 3.132020509610811e-08, 3.5748260529544496e-07, 6.648343742199359e-08, 3.163856945320731e-07, 1.2095728152416996e-06, 8.663983663836916e-08, 4.014367860349921e-08, 1.2544606420306081e-07, 3.320077013313494e-08, 1.973121896980956e-07, 3.8063546980993124e-07, 1.382770449254167e-07, 2.3001977211833946e-08, 7.14414980507172e-08, 2.851079372589993e-08, 2.982524804906461e-08, 8.935905526641363e-08, 6.223806394700659e-07, 6.478746428228987e-08, 1.3662854314588913e-07, 9.973038572752557e-08, 2.576123669939534e-08, 4.130169273253159e-08, 3.9383902361578293e-08, 1.2157941853274679e-07, 4.302776233089389e-06, 1.8178942582380841e-06, 1.8520014464229462e-06, 1.6245859342234326e-06, 1.64481571118813e-05, 1.0544756150920875e-05, 3.993004611402284e-06, 2.646454788646224e-07, 1.919327405630611e-05, 4.803187607649306e-07, 1.6962640358997305e-07, 4.550436642603017e-06, 4.2358526116004214e-05, 4.443309080670588e-06, 5.103951934870565e-06, 1.3506267350749113e-05, 4.1758286783988297e-07, 4.4944314936401497e-07, 3.156656873670727e-07, 1.0557428140600678e-06, 1.3364497952750298e-08, 1.3893487249561076e-08, 6.770295613023336e-08, 1.4129548731034447e-07, 7.170429228153807e-08, 7.934354329108828e-08, 2.6391164809069778e-08, 2.6133946562367782e-08, 7.1966232972897615e-09, 2.1752097012495142e-08, 6.684530973188885e-08, 3.4177467966856057e-08, 1.6228092647452286e-07, 4.107051836399478e-07, 6.472068889706861e-07, 2.9513313393181306e-07, 5.653383094994524e-09, 4.8300755395302986e-08, 8.887371727439586e-09, 3.7305639466467255e-08, 1.7784008932153483e-08, 4.641856232723285e-09, 3.413062898971475e-08, 1.9373679549516964e-07, 1.2980041219634586e-06, 3.5640496776068176e-08, 2.1493017143825455e-08, 3.055253330330743e-07, 1.553226809392072e-07, 1.4520828699460253e-06, 3.488421285169352e-08, 3.825387466349639e-05, 4.508809183789708e-07, 4.176648928932991e-07, 6.770537765987683e-07, 1.4142071336209483e-07, 1.423571211489616e-05, 6.293736873885791e-07, 4.762812295666663e-06, 9.024806217894366e-07, 9.058892942448438e-07, 1.5713549146312289e-06, 1.5720408441666223e-07, 1.8185124872616143e-07, 7.193072093514274e-08, 1.1952631666645175e-06, 8.874753234522359e-07, 2.0870552930318809e-07, 9.906116105184992e-08, 7.793509837483725e-09, 1.0058372623689138e-07, 4.205883783470199e-07, 1.8436001880672848e-07, 1.6437729755125474e-07, 7.02525184692604e-08, 2.5704193262754416e-07, 7.586770323086967e-08, 7.841231308702845e-07, 2.495280568837188e-07, 5.157617977147311e-08, 4.0673629797538524e-08, 7.531674839356128e-09, 4.797425745550754e-08, 1.7419722553313477e-08, 1.7958475950763386e-07, 1.2566235163546935e-08, 8.975329990335013e-08, 3.269607518063822e-08, 1.1208221195602164e-07, 3.906690437815996e-08, 4.676846998563633e-08, 1.852334321483795e-07, 1. [e2e-predictor] 4832883721282997e-07, 1.2279221550670627e-07, 1.0729002042353386e-06, 3.6537970427019673e-09, 1.6198204377815273e-07, 1.6190487173162182e-08, 1.2004819893718377e-07, 1.480009448329156e-08, 4.022906807676918e-08, 2.1505718450498534e-07, 1.192555032503151e-07, 4.898190297808469e-08, 7.608841201545147e-08, 2.3137237903370078e-08, 8.52094430570105e-08, 9.586112525994395e-08, 1.3351485961266008e-07, 3.0216618540634954e-08, 4.4238134933038964e-08, 2.610634552979718e-08, 2.397684113475407e-07, 1.3192399705985736e-07, 1.6734691854480843e-08, 1.5883161097463017e-07, 4.064300753725547e-07, 8.753363545110915e-08, 8.366871497855755e-07, 3.437562767771851e-08, 7.847781802183817e-08, 8.526320982582547e-09, 9.601245665180613e-08, 5.258879127723048e-07, 1.3557291822507977e-07, 1.0307118003538562e-07, 1.0429684671464656e-08, 5.187630236491714e-08, 2.1869702493404475e-08, 1.179131725592697e-08, 2.9806198398318884e-08, 4.3383401759911067e-08, 2.999067660880428e-08, 2.8507258775789523e-08, 3.058627484620047e-08, 6.441018740588333e-08, 1.5363882610586188e-08, 1.597368459727022e-08, 2.57365329048298e-08, 1.090363141997841e-07, 3.211827603877282e-08, 6.819658615597746e-09, 1.9251054084179486e-07, 5.825749838095362e-08, 1.876556723345857e-07, 4.0070369777822634e-07, 1.5791380647556252e-08, 1.9501307235714194e-07, 1.0141919304373914e-08, 2.744775962071344e-08, 1.2843395680306458e-08, 3.7296960186949946e-08, 7.407404467585366e-08, 4.206033921150265e-08, 1.6924593282396927e-08, 1.4591853414458456e-07, 4.344931525679385e-08, 1.7191209167322086e-07, 3.5816999144344663e-08, 8.440128596021168e-09, 4.1947775031303536e-08, 2.513996122388562e-08, 2.8339847801817086e-08, 8.747072399728495e-08, 8.277008411994302e-09, 1.1676148936601294e-08, 1.4548319882123906e-07, 7.200192886358536e-09, 2.623576165206032e-06, 5.675644842995098e-07, 1.9483265987219056e-06, 6.752511438889996e-08, 8.168359499904909e-08, 1.0932910754490877e-07, 1.6708914074570203e-07, 3.138676518688044e-08, 2.9734879447573803e-08, 5.7520821883372264e-08, 5.8507939826313304e-08, 3.2544284067625995e-07, 3.330167075432655e-08, 4.186742614820105e-07, 1.5085691984495497e-07, 2.334644193524582e-07, 2.8639897209359333e-07, 2.2940034227758588e-07, 1.8537357959758083e-07, 3.151756686747831e-07, 1.1075776455982123e-06, 1.5369606387594104e-07, 1.9237216974943294e-07, 1.6404264613356645e-07, 2.900799245253438e-07, 1.2466382770526252e-07, 5.802550617772795e-08, 5.1861558603150115e-08, 6.009345909774311e-09, 1.23330778478703e-07, 1.3798263864828186e-07, 1.7370017246776115e-07, 5.617696388071636e-07, 5.160393001801822e-08, 4.813227150179955e-08, 8.032599652096906e-08, 2.0644911273848265e-06, 5.638538027596951e-07, 8.794099244369136e-07, 3.4785582556651207e-06, 2.972355162000895e-07, 3.390414349269122e-07, 9.468957529179534e-08, 3.754799493549399e-08, 1.5678830322940485e-07, 8.2030041426151e-08, 6.847877820348458e-09, 1.825139683830912e-08, 6.050154155445853e-08, 3.9562941367421445e-08, 1.0699794472657231e-07, 3.256623699599004e-07, 3.536899839673424e-07, 7.326217854597417e-08, 4.857605517827324e-07, 7.717640073678922e-07, 3.456741737295488e-08, 3.2461647947457095e-07, 3.1608240078639938e-06, 5.330938535053065e-08, 3.645648973815696e-07, 5.481521156980307e-07, 4.623012372917401e-08, 1.346600981833035e-07, 4.352903104631878e-08, 1.6403919289587066e-07, 2.4636646500653114e-08, 5.958662185889807e-08, 9.493569308460792e-08, 5.5234050222452424e-08, 5.741187081298449e-08, 1.1850225746457e-05, 5.826300366607029e-06, 7.420818292303011e-06, 9.127862199420633e-07, 2.001935399675858e-06, 1.0334889566365746e-06, 3.514648483360361e-08, 2.0589775431290036e-06, 3.5655068586493144e-07, 6.873136015883574e-08, 2.193502490754895e-09, 5.560284321859399e-08, 3.3266616128457827e-07, 1.3073517379780242e-07, 2.7187292417352182e-08, 1.0462799515664756e-08, 7.466601914529747e-07, 6.923080064780152e-08, 1.6145493830777013e-08, 8.568415310605815e-09, 4.762142236103273e-09, 1.233958357715892e-07, 8.340525603500737e-09, 3.2648699299642203e-09, 5.063196084620358e-09, 4.0704488668552585e-09, 1.2043389396865223e-08, 5.1055555339019065e-09, 7 [e2e-predictor] .267066948202228e-09, 1.1844983305309142e-07, 7.53547197973603e-08, 6.386873252495207e-08, 1.621267564644313e-08, 2.6103893446816073e-07, 6.979319664424111e-07, 6.647392325476176e-08, 7.717375183347031e-07, 6.651137596236367e-07, 3.3244478458982485e-07, 3.707229154770175e-07, 3.995585302618565e-07, 6.411539743567118e-08, 7.107263400030206e-08, 1.6379799205878953e-07, 6.876715730186334e-08, 3.4624252975845593e-07, 2.0256192101442139e-07, 6.19600768914097e-07, 2.6840686118134727e-08, 6.720200644849683e-07, 1.1348156476742588e-06, 1.8397651047052932e-06, 6.397185075002199e-07, 7.257429501805746e-08, 4.2213261508550204e-07, 3.965735402289283e-07, 1.4037237860975438e-07, 3.249813573802385e-07, 1.5857477819736232e-07, 1.1122037335553614e-07, 7.39131564841955e-08, 3.4231769063808315e-07, 5.39788445053091e-08, 8.517191218970765e-08, 4.060975243191933e-06, 1.4478493540082127e-05, 7.316928396505773e-09, 6.948383646232514e-09, 4.4688626132938225e-08, 9.23126250995665e-08, 5.41190452452156e-08, 2.224259958438779e-07, 1.7609368185844687e-08, 2.0905980946395175e-08, 3.6797160873902612e-09, 6.177843658861093e-08, 1.7920085326750268e-07, 2.6278852871541858e-08, 2.698776313536655e-08, 1.6432605320915172e-07, 1.2827440798446332e-07, 4.468845560268164e-08, 6.316474099321567e-08, 1.9461522171582146e-08, 2.0871000927513705e-08, 2.2414262446091016e-08, 2.4764982953229264e-08, 6.785314532464781e-09, 2.4248494767675766e-08, 4.554914667664889e-09, 2.8976625543464252e-08, 2.0402152856036082e-08, 1.6009290959573264e-07, 2.019951637066697e-07, 1.6111656009343278e-08, 5.964040639128143e-08, 4.0982603977113286e-09, 3.907972612182675e-08, 7.511245847524606e-09, 5.965555942566425e-07, 1.6478051634294388e-07, 1.410681527147517e-08, 3.285524030616216e-08, 3.338656018314623e-09, 1.2200259469352659e-08, 4.624581961820695e-08, 6.842250765970448e-09, 1.7426071252657493e-08, 4.661075436729334e-08, 9.33183343931887e-08, 1.230591379908219e-07, 1.2359244294657401e-08, 1.1173082903326303e-08, 2.772451956900568e-08, 2.4191731995415466e-07, 3.4511506896706123e-07, 2.593734649281032e-08, 9.964443847820803e-08, 9.797671118860762e-09, 1.90853270964908e-07, 3.9726497647052383e-08, 2.6638923955601967e-08, 6.8740502179309715e-09, 3.146955052102385e-08, 2.4086276084744895e-07, 1.3126738451774145e-07, 2.1254035686979478e-07, 2.0501781605730685e-08, 3.694923123020999e-08, 6.563101351275691e-07, 2.5600135344916453e-08, 2.688264899575188e-08, 6.880517844365386e-07, 2.0092501529234141e-07, 2.7880052044793047e-08, 2.62837644982028e-08, 5.167780159354152e-08, 1.893516099471526e-07, 4.6184757707123936e-07, 1.1086663143089481e-08, 1.4542384008109366e-07, 2.8736709012378014e-08, 6.105074703555147e-07, 1.2016337258557996e-08, 1.3944533350240818e-07, 2.0936788303060894e-08, 4.3800788773751265e-08, 1.9662968142597492e-08, 6.6543535126584175e-09, 1.1149432488366529e-08, 6.424846787922434e-08, 6.971495380980741e-09, 3.2599853927450795e-09, 1.4260012726197147e-08, 2.7894840215481054e-08, 8.115682703646598e-08, 2.599534454361674e-08, 2.285521105704902e-08, 1.05594978094814e-07, 8.145473628928812e-08, 3.779314994289962e-08, 4.8818307618603285e-08, 2.3420332340151617e-08, 1.0599223720930695e-08, 3.604067799756194e-08, 5.062377184117395e-08, 3.6804056691153164e-08, 1.5015616527946918e-09, 1.4474869658442913e-06, 1.0761959856608883e-06, 1.304976251503831e-07, 3.073563448197092e-08, 1.7183838352252678e-08, 2.042087920983704e-08, 7.904924714807748e-09, 1.6901843480354728e-07, 1.2416376371504612e-08, 5.4757610001843204e-08, 2.6250097207025647e-08, 1.3261202802539174e-08, 6.298007804161898e-08, 1.2709827856838274e-08, 2.0171410142211244e-07, 4.386575014336813e-08, 6.962276444255622e-08, 2.956475668725034e-07, 7.925040677037032e-07, 2.0868660044470744e-07, 1.7341581326490996e-07, 4.294189182019181e-08, 4.213362547744737e-09, 8.8246935092684e-08, 1.7341355018629656e-08, 7.321497008661026e-08, 1.7941481189609476e-08, 1.1245008835203407e-07, 4.2423405943736725e-07, 8.259470440918903e-09, 1.1336252470073305e-07, 8.268681739309613e-08, 2.2186704029536486e-08, 1.9539518447686532e-08, 1.0675572426066537e [e2e-predictor] -08, 3.288476690954667e-08, 2.4340661042288048e-08, 6.639029948019015e-08, 5.604618547039308e-09, 1.386590220420203e-08, 6.6757905869963e-08, 1.1355724360839758e-08, 3.1321272331297223e-07, 3.1244848486267074e-08, 1.502163371469578e-07, 1.346161226933873e-08, 1.8882738572756352e-07, 4.645693962856967e-08, 4.645321638463429e-08, 7.71435715307689e-09, 3.585664032357272e-08, 7.608256247237932e-09, 4.2214487905312126e-08, 4.3406713778892936e-09, 1.3401408871516196e-08, 8.565495335233209e-08, 1.7045202582721686e-08, 5.422122839604526e-08, 3.0218750168842234e-08, 6.153300091682468e-08, 3.9388163841636015e-09, 4.135585029985123e-08, 1.7818981845607595e-08, 4.31053557292671e-08, 3.9032919119108556e-09, 7.663005163749403e-08, 1.1890258555524724e-08, 9.304103265606045e-09, 1.0967905472725192e-09, 1.0536636452229686e-08, 1.1516662112853737e-07, 8.134422273542441e-07, 5.9525618212319387e-08, 2.8063100288022724e-08, 1.283296402476708e-08, 1.060557863752365e-07, 7.872866945035639e-07, 2.7501052102252288e-08, 3.936232229051484e-09, 2.0519189902756807e-08, 7.394696410756296e-09, 3.5983390489491285e-08, 2.5378296797384792e-08, 4.6989228508209635e-08, 7.549436631393291e-09, 6.322734407149255e-07, 5.581916262542563e-09, 1.2963950268840563e-07, 1.587476106124086e-08, 3.3837391555380236e-08, 6.474394353972457e-09, 9.121036015358186e-08, 1.3918366015275296e-08, 8.229938508463874e-09, 2.7060956142577197e-08, 2.6095642979839795e-08, 5.722282203635132e-09, 6.963402938708896e-07, 4.685033871965061e-08, 9.590480409826796e-09, 2.0991818416860042e-07, 3.082116961650172e-08, 3.563118511351604e-08, 7.326215154535021e-07, 2.1759417450084584e-06, 2.4074907400972734e-07, 2.974484800688515e-07, 2.528977205429328e-08, 7.667869894589785e-09, 2.6632636718204594e-07, 3.435845030708151e-08, 2.3129958393042216e-08, 3.1239306252928145e-08, 2.8691238185274415e-07, 3.8957892911639647e-08, 2.4183783864373254e-08, 1.158230222841894e-08, 5.1544706280992614e-08, 2.0343204454320585e-08, 8.201909196259294e-08, 1.1641392738681589e-08, 5.496298740581551e-07, 1.1682007361457636e-08, 4.7576321549058775e-08, 1.6349654075042963e-08, 4.090812311119407e-08, 2.1270925287808495e-07, 1.669703095785735e-07, 3.989651631286506e-08, 2.8524043500510743e-06, 1.2500227342115977e-07, 2.4846258384059183e-07, 1.2454135500661323e-08, 2.9699904757762852e-08, 4.25093871214699e-09, 1.6144196024470148e-07, 2.6385819751340023e-07, 7.637975585339518e-09, 3.4455236441743864e-09, 7.273212787595185e-08, 1.763124757303558e-08, 7.586505645917896e-09, 2.154678924171094e-08, 1.267521554382256e-07, 2.5636829548147944e-08, 3.500932521660616e-08, 6.472151170555662e-08, 8.387794458997178e-09, 3.0694742747527926e-08, 7.520281286588215e-08, 1.5724816648798878e-07, 1.9633705505839316e-07, 1.2290654183289007e-07, 1.1121071841202479e-09, 1.5468767600168576e-08, 9.916888643601851e-09, 6.882375487293757e-07, 8.267530660077682e-08, 4.46946799570469e-08, 2.075171856574798e-08, 8.649272587035739e-08, 5.20271186132959e-08, 4.5564465978031876e-08, 2.031970502969216e-08, 8.705073994974555e-09, 6.45198596771479e-08, 2.1777212921847422e-08, 1.0309374687267336e-08, 3.2118705917127954e-08, 2.3336649235261575e-07, 8.053995692591798e-09, 1.9623111313649133e-07, 1.2887916511772346e-07, 1.5392334518082862e-08, 1.401880500573327e-09, 5.696758975659577e-08, 6.079961423210989e-09, 1.0782660986308201e-08, 2.4260382147645032e-08, 1.9388348704296732e-08, 2.2969983604070876e-07, 1.997151066746028e-08, 2.8477696645268225e-08, 5.227270349905666e-08, 2.739241153904004e-07, 9.85714976309282e-08, 3.129077086327925e-08, 4.1513992243835673e-08, 5.2511412995670526e-09, 1.5806674582563573e-06, 8.547505672140687e-07, 1.0688979479311911e-08, 1.0621677120070672e-06, 7.737217089243131e-08, 6.394137130882882e-08, 1.1698201518584028e-07, 1.0486479595783749e-07, 2.1160776952910965e-07, 1.5339432835048683e-08, 5.094390331805698e-08, 1.4005206772083056e-08, 2.6281709253339613e-08, 8.77830927947798e-08, 7.771985721660712e-09, 4.228822803042931e-08, 3.3243432540075446e-07, 7.72913466562386e-08, 7.63679175452836e-10, 5.989415541307608e-08, 1.32607539171658 [e2e-predictor] 57e-07, 1.2853475084284582e-07, 8.844149412823299e-09, 1.0194228394766469e-07, 2.4937534703894926e-07, 1.6547782877296413e-07, 1.176258734858493e-08, 1.1496064900029523e-07, 2.934237670615403e-07, 1.3261050924029405e-08, 8.630173198298507e-08, 5.739413211358624e-08, 1.1093964502606468e-07, 2.2933429022486962e-07, 3.470581191322708e-07, 1.475092403779854e-07, 1.5024763655446804e-08, 6.454251177956394e-08, 5.164459437878577e-08, 6.23732674398525e-08, 1.2935890936205396e-07, 1.4051898311606692e-08, 5.386868906498421e-08, 2.082729722019394e-08, 1.3040525104202061e-08, 1.05788906523685e-07, 1.507953939494655e-08, 8.926216423787992e-07, 4.637372086335745e-08, 7.480928161385236e-07, 5.883074472023964e-09, 2.870727655590599e-09, 8.381493898923509e-07, 7.341868002441743e-09, 1.4245849833116608e-08, 1.0926260785026898e-07, 1.1308060265946551e-07, 2.52335809136639e-07, 1.178269073420779e-07, 4.667730024721095e-08, 2.79587997198405e-09, 3.436349871321909e-08, 1.4674341741738317e-07, 3.539640758276619e-08, 2.0580911552769976e-07, 7.18380377406902e-08, 2.7693602078215918e-08, 4.5492825506698864e-08, 1.955898198957584e-08, 1.5359576721607482e-08, 1.2336422550163206e-08, 2.957029643368969e-08, 2.877517033539334e-07, 9.051751135302766e-07, 2.3732353326977318e-07, 1.6521470413977113e-08, 1.5478654802336678e-08, 3.526755776306345e-08, 3.61636587342673e-08, 1.615889466677345e-08, 7.649993705172164e-08, 1.9661243300106435e-08, 4.9174737171142624e-08, 1.1712456426948847e-07, 1.0889120716228717e-08, 1.4941074368834961e-06, 1.0185750909386115e-08, 3.7575578204496196e-08, 2.0970718495050278e-08, 3.368522527580353e-08, 4.845519985252622e-09, 6.039555842107802e-07, 1.0373183201295433e-08, 2.841620698745828e-07, 4.499007388858445e-07, 3.4631430168019506e-08, 7.720588968140873e-08, 1.4711066853578814e-07, 1.160143270340086e-07, 4.0074306184578745e-07, 3.0256124716743216e-08, 6.706726196625823e-08, 2.012845534693497e-08, 1.5987545731732666e-09, 4.1919221871467016e-08, 1.3166965295852151e-08, 3.231780709711529e-08, 9.247563781400459e-08, 1.3075164417841734e-07, 1.0574149911235509e-07, 3.762125899697821e-08, 1.0942090256094161e-07, 7.001387558602801e-08, 2.7426720450307585e-08, 2.0766329100752046e-08, 4.5402845927355884e-08, 3.390360703292572e-08, 1.0469531730450399e-07, 2.8271228913467894e-08, 3.406190955956845e-07, 5.146158059687878e-07, 6.740644380442973e-07, 6.3821934404018066e-09, 3.638748680145909e-08, 3.6260146885069844e-08, 1.6065344254911906e-07, 3.639007957190188e-07, 6.232625970881145e-09, 4.8054303647404595e-08, 3.3725854109434295e-08, 6.3287848206528e-07, 6.480550496235082e-08, 2.116489525860743e-07, 8.38768769995113e-08, 1.758889212055692e-08, 2.729004577872729e-09, 2.144768451728396e-08, 7.861175532752895e-08, 2.011793931444572e-08, 2.840733515085958e-08, 2.492219834948628e-07, 2.0156347702027233e-08, 2.6551370879701608e-08, 2.7847898209643063e-08, 6.907038496706264e-09, 1.8805204504701578e-08, 1.3006692611838844e-08, 3.685865976876812e-07, 3.9678877783444477e-07, 2.7591790185965692e-08, 2.5228539257682314e-08, 1.5469838388071366e-07, 3.6892611632310945e-08, 1.4401622117077295e-09, 2.1504664715621402e-08, 5.068058328561165e-08, 5.081657761252245e-08, 1.1718628734058711e-08, 5.409200909411993e-08, 7.138175419640902e-07, 2.523685509459028e-07, 7.071956531490287e-08, 7.199660245760242e-08, 1.2525339876390262e-08, 3.4802988579940575e-07, 1.9591639954796847e-07, 1.2404500182583433e-07, 1.2346009725661133e-07, 1.92011366806355e-08, 1.9895020386684337e-07, 3.7876212388709973e-08, 1.0629652713589621e-08, 1.2436973229057458e-08, 2.186162078032794e-07, 2.618108396745811e-07, 1.1128912547064829e-07, 1.0776497560982534e-07, 6.380246109216614e-09, 3.895029632161595e-09, 1.576254788915321e-07, 2.9089968212048234e-09, 1.03809227880447e-08, 2.8134863327977655e-08, 1.0777869263733919e-08, 1.3605805371241786e-08, 2.9236161935841665e-08, 1.918957508451058e-07, 2.1994748067299952e-07, 1.3263833587018325e-08, 4.900423533626963e-08, 2.9808058243929736e-09, 8.925934658066126e-09, 1.0996840771326788e-08, 7.715520666806697e-09, 7.454385553273823e-09, 5.086099008 [e2e-predictor] 6221136e-08, 1.5129604946650943e-07, 1.1959995660504319e-08, 1.1323217030678734e-08, 9.39119804144184e-09, 9.585582461113518e-08, 1.9055129030220996e-08, 1.8105044574667772e-08, 6.179151057494892e-08, 6.389322493305372e-08, 1.1853056314237165e-08, 9.372677745034252e-09, 1.2332006349424773e-07, 1.6521848777983905e-08, 1.246099685658919e-07, 4.196111902388111e-09, 3.9965367903960214e-08, 1.255440107428285e-08, 1.4302015571843185e-08, 6.631698745707126e-09, 5.964162053118116e-09, 5.556878956980427e-09, 1.419230528654225e-07, 1.7612896030527736e-08, 3.3801444487835397e-07, 7.856438344333583e-08, 2.9667463152804885e-08, 2.8991748877160717e-06, 1.3787196166958893e-06, 5.313577133847502e-09, 2.5128217728820346e-08, 8.406522056247923e-08, 4.492775573794461e-08, 5.4087468726038423e-08, 2.4238921980668238e-08, 4.0167356019082945e-07, 4.1083040969169815e-08, 5.415281023601892e-08, 8.640659743264223e-09, 5.773174294176897e-08, 2.6442751277500065e-07, 8.953106203080097e-07, 2.799365539374321e-08, 5.889395637836969e-09, 4.178830081968954e-08, 2.888028660663622e-08, 2.1357223545237503e-08, 2.3023956430279213e-07, 8.786517469161481e-08, 2.069703963769598e-09, 2.2363829899063603e-08, 3.2032425600903025e-09, 1.176860120466472e-08, 6.963498577761129e-08, 2.2719042647167953e-08, 7.360319465021803e-09, 6.922469264480924e-09, 3.2138252947788715e-08, 1.3705631829452614e-07, 1.9814805796158907e-08, 1.0578844467090676e-08, 2.7049035011827982e-08, 2.9755513608620277e-09, 2.1773475467057324e-08, 1.0975404585167325e-08, 1.9918440230526357e-08, 2.3881801425318372e-08, 2.1357100976615584e-08, 6.109046157831699e-09, 1.1890372242362446e-08, 1.1459750481890296e-08, 3.731681275098708e-08, 1.5726065782928345e-08, 3.403987491878979e-08, 3.6921193213856895e-08, 9.281668411631472e-08, 1.2323060616381554e-07, 4.234707162709128e-08, 1.7423577247654976e-08, 5.811278214196136e-08, 3.9313878374969136e-08, 2.3690166273127033e-08, 1.792834680713895e-08, 1.4406491999352511e-08, 7.0197305745978156e-09, 6.041436506620812e-08, 4.867428771149207e-08, 1.0685186957459791e-08, 1.005112082452797e-08, 4.2425732260653604e-08, 2.6077879411445792e-08, 5.136626679558276e-09, 1.6972753202892932e-09, 1.9131350725842822e-08, 2.1117190840413969e-07, 1.3372530638378066e-08, 2.000219723186092e-08, 1.0454146348592985e-07, 2.814382682458927e-08, 2.134452614654947e-07, 2.104584950757271e-08, 1.9114216343041335e-07, 3.957645233754192e-08, 2.9315843974586642e-08, 1.1051893977764848e-07, 4.840025980001883e-08, 5.583537188158516e-08, 1.2129960680340446e-07, 1.7761975001917563e-08, 2.561053769056798e-08, 1.2038481145282276e-07, 4.6748018789344314e-09, 1.591587306393194e-08, 3.147561500327356e-08, 6.147622144681009e-08, 2.204614979461894e-08, 3.2577264619249036e-07, 1.198899752807847e-07, 2.3818718375423487e-07, 1.490982981522393e-08, 5.1016343149967724e-08, 5.5141523347401744e-08, 2.288526701477167e-08, 5.7140397302646306e-08, 5.185041231925425e-07, 4.977233558633998e-08, 1.104976110610778e-08, 1.26408238543263e-07, 8.174780674607973e-08]]} [e2e-predictor] 2026-07-28 17:18:57.737 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [[6.736569968524009e-09, 1.153582651625129e-08, 5.142186410012073e-08, 6.647101002954514e-08, 4.094442118685038e-08, 1.3402272713847196e-07, 9.35544619551365e-08, 2.8935589924117266e-08, 6.845294819868286e-08, 7.680520752728626e-08, 2.0334418877610005e-06, 1.1085519417974865e-06, 2.3477346644540376e-07, 6.581955744877632e-07, 0.00012372950732242316, 4.2878312456196e-07, 6.419879809982376e-06, 0.9993487000465393, 7.371911488007754e-05, 3.101079710177146e-05, 5.602713827101979e-06, 2.1862256289750803e-06, 1.9543842100233633e-08, 3.728848128048412e-07, 4.29031416615544e-07, 1.8250955235998845e-07, 7.159837878134567e-08, 9.231503561579757e-09, 6.469168170042394e-07, 7.031603299623157e-09, 4.4511686780879245e-08, 1.2455829789814743e-07, 9.446210924579646e-08, 4.3476589439706004e-08, 4.6581672563661414e-07, 6.797636586952649e-08, 2.1060014887552825e-07, 2.2604888627597575e-08, 1.4311326879123953e-07, 7.951216218771151e-08, 1.2341666888460168e-07, 1.0921577313638409e-06, 1.5243762391037308e-05, 3.1172930903267115e-07, 2.415180802017858e-07, 6.863702850523623e-08, 8.46760883632669e-08, 9.424560687421035e-08, 1.0219084245477461e-08, 3.3770398033539095e-08, 3.6928447855189006e-08, 1.3693848188722768e-07, 1.067415098532365e-07, 2.5994486918534676e-07, 3.486601087843155e-07, 3.132020509610811e-08, 3.5748260529544496e-07, 6.648343742199359e-08, 3.163856945320731e-07, 1.2095728152416996e-06, 8.663983663836916e-08, 4.014367860349921e-08, 1.2544606420306081e-07, 3.320077013313494e-08, 1.973121896980956e-07, 3.8063546980993124e-07, 1.382770449254167e-07, 2.3001977211833946e-08, 7.14414980507172e-08, 2.851079372589993e-08, 2.982524804906461e-08, 8.935905526641363e-08, 6.223806394700659e-07, 6.478746428228987e-08, 1.3662854314588913e-07, 9.973038572752557e-08, 2.576123669939534e-08, 4.130169273253159e-08, 3.9383902361578293e-08, 1.2157941853274679e-07, 4.302776233089389e-06, 1.8178942582380841e-06, 1.8520014464229462e-06, 1.6245859342234326e-06, 1.64481571118813e-05, 1.0544756150920875e-05, 3.993004611402284e-06, 2.646454788646224e-07, 1.919327405630611e-05, 4.803187607649306e-07, 1.6962640358997305e-07, 4.550436642603017e-06, 4.2358526116004214e-05, 4.443309080670588e-06, 5.103951934870565e-06, 1.3506267350749113e-05, 4.1758286783988297e-07, 4.4944314936401497e-07, 3.156656873670727e-07, 1.0557428140600678e-06, 1.3364497952750298e-08, 1.3893487249561076e-08, 6.770295613023336e-08, 1.4129548731034447e-07, 7.170429228153807e-08, 7.934354329108828e-08, 2.6391164809069778e-08, 2.6133946562367782e-08, 7.1966232972897615e-09, 2.1752097012495142e-08, 6.684530973188885e-08, 3.4177467966856057e-08, 1.6228092647452286e-07, 4.107051836399478e-07, 6.472068889706861e-07, 2.9513313393181306e-07, 5.653383094994524e-09, 4.8300755395302986e-08, 8.887371727439586e-09, 3.7305639466467255e-08, 1.7784008932153483e-08, 4.641856232723285e-09, 3.413062898971475e-08, 1.9373679549516964e-07, 1.2980041219634586e-06, 3.5640496776068176e-08, 2.1493017143825455e-08, 3.055253330330743e-07, 1.553226809392072e-07, 1.4520828699460253e-06, 3.488421285169352e-08, 3.825387466349639e-05, 4.508809183789708e-07, 4.176648928932991e-07, 6.770537765987683e-07, 1.4142071336209483e-07, 1.423571211489616e-05, 6.293736873885791e-07, 4.762812295666663e-06, 9.024806217894366e-07, 9.058892942448438e-07, 1.5713549146312289e-06, 1.5720408441666223e-07, 1.8185124872616143e-07, 7.193072093514274e-08, 1.1952631666645175e-06, 8.874753234522359e-07, 2.0870552930318809e-07, 9.906116105184992e-08, 7.793509837483725e-09, 1.0058372623689138e-07, 4.205883783470199e-07, 1.8436001880672848e-07, 1.6437729755125474e-07, 7.02525184692604e-08, 2.5704193262754416e-07, 7.586770323086967e-08, 7.841231308702845e-07, 2.495280568837188e-07, 5.157617977147311e-08, 4.0673629797538524e-08, 7.531674839356128e-09, 4.797425745550754e-08, 1.7419722553313477e-08, 1.7958475950763386e-07, 1.2566235163546935e-08, 8.975329990335013e-08, 3.269607518063822e-08, 1.1208221195602164e-07, 3.906690437815996e-08, 4.676846998563633e [e2e-predictor] -08, 1.852334321483795e-07, 1.4832883721282997e-07, 1.2279221550670627e-07, 1.0729002042353386e-06, 3.6537970427019673e-09, 1.6198204377815273e-07, 1.6190487173162182e-08, 1.2004819893718377e-07, 1.480009448329156e-08, 4.022906807676918e-08, 2.1505718450498534e-07, 1.192555032503151e-07, 4.898190297808469e-08, 7.608841201545147e-08, 2.3137237903370078e-08, 8.52094430570105e-08, 9.586112525994395e-08, 1.3351485961266008e-07, 3.0216618540634954e-08, 4.4238134933038964e-08, 2.610634552979718e-08, 2.397684113475407e-07, 1.3192399705985736e-07, 1.6734691854480843e-08, 1.5883161097463017e-07, 4.064300753725547e-07, 8.753363545110915e-08, 8.366871497855755e-07, 3.437562767771851e-08, 7.847781802183817e-08, 8.526320982582547e-09, 9.601245665180613e-08, 5.258879127723048e-07, 1.3557291822507977e-07, 1.0307118003538562e-07, 1.0429684671464656e-08, 5.187630236491714e-08, 2.1869702493404475e-08, 1.179131725592697e-08, 2.9806198398318884e-08, 4.3383401759911067e-08, 2.999067660880428e-08, 2.8507258775789523e-08, 3.058627484620047e-08, 6.441018740588333e-08, 1.5363882610586188e-08, 1.597368459727022e-08, 2.57365329048298e-08, 1.090363141997841e-07, 3.211827603877282e-08, 6.819658615597746e-09, 1.9251054084179486e-07, 5.825749838095362e-08, 1.876556723345857e-07, 4.0070369777822634e-07, 1.5791380647556252e-08, 1.9501307235714194e-07, 1.0141919304373914e-08, 2.744775962071344e-08, 1.2843395680306458e-08, 3.7296960186949946e-08, 7.407404467585366e-08, 4.206033921150265e-08, 1.6924593282396927e-08, 1.4591853414458456e-07, 4.344931525679385e-08, 1.7191209167322086e-07, 3.5816999144344663e-08, 8.440128596021168e-09, 4.1947775031303536e-08, 2.513996122388562e-08, 2.8339847801817086e-08, 8.747072399728495e-08, 8.277008411994302e-09, 1.1676148936601294e-08, 1.4548319882123906e-07, 7.200192886358536e-09, 2.623576165206032e-06, 5.675644842995098e-07, 1.9483265987219056e-06, 6.752511438889996e-08, 8.168359499904909e-08, 1.0932910754490877e-07, 1.6708914074570203e-07, 3.138676518688044e-08, 2.9734879447573803e-08, 5.7520821883372264e-08, 5.8507939826313304e-08, 3.2544284067625995e-07, 3.330167075432655e-08, 4.186742614820105e-07, 1.5085691984495497e-07, 2.334644193524582e-07, 2.8639897209359333e-07, 2.2940034227758588e-07, 1.8537357959758083e-07, 3.151756686747831e-07, 1.1075776455982123e-06, 1.5369606387594104e-07, 1.9237216974943294e-07, 1.6404264613356645e-07, 2.900799245253438e-07, 1.2466382770526252e-07, 5.802550617772795e-08, 5.1861558603150115e-08, 6.009345909774311e-09, 1.23330778478703e-07, 1.3798263864828186e-07, 1.7370017246776115e-07, 5.617696388071636e-07, 5.160393001801822e-08, 4.813227150179955e-08, 8.032599652096906e-08, 2.0644911273848265e-06, 5.638538027596951e-07, 8.794099244369136e-07, 3.4785582556651207e-06, 2.972355162000895e-07, 3.390414349269122e-07, 9.468957529179534e-08, 3.754799493549399e-08, 1.5678830322940485e-07, 8.2030041426151e-08, 6.847877820348458e-09, 1.825139683830912e-08, 6.050154155445853e-08, 3.9562941367421445e-08, 1.0699794472657231e-07, 3.256623699599004e-07, 3.536899839673424e-07, 7.326217854597417e-08, 4.857605517827324e-07, 7.717640073678922e-07, 3.456741737295488e-08, 3.2461647947457095e-07, 3.1608240078639938e-06, 5.330938535053065e-08, 3.645648973815696e-07, 5.481521156980307e-07, 4.623012372917401e-08, 1.346600981833035e-07, 4.352903104631878e-08, 1.6403919289587066e-07, 2.4636646500653114e-08, 5.958662185889807e-08, 9.493569308460792e-08, 5.5234050222452424e-08, 5.741187081298449e-08, 1.1850225746457e-05, 5.826300366607029e-06, 7.420818292303011e-06, 9.127862199420633e-07, 2.001935399675858e-06, 1.0334889566365746e-06, 3.514648483360361e-08, 2.0589775431290036e-06, 3.5655068586493144e-07, 6.873136015883574e-08, 2.193502490754895e-09, 5.560284321859399e-08, 3.3266616128457827e-07, 1.3073517379780242e-07, 2.7187292417352182e-08, 1.0462799515664756e-08, 7.466601914529747e-07, 6.923080064780152e-08, 1.6145493830777013e-08, 8.568415310605815e-09, 4.762142236103273e-09, 1.233958357715892e-07, 8.340525603500737e-09, 3.2648699299642203e-09, 5.063196084620358e-09, 4.0704488668552585e-09, 1.2043389396865223e [e2e-predictor] -08, 5.1055555339019065e-09, 7.267066948202228e-09, 1.1844983305309142e-07, 7.53547197973603e-08, 6.386873252495207e-08, 1.621267564644313e-08, 2.6103893446816073e-07, 6.979319664424111e-07, 6.647392325476176e-08, 7.717375183347031e-07, 6.651137596236367e-07, 3.3244478458982485e-07, 3.707229154770175e-07, 3.995585302618565e-07, 6.411539743567118e-08, 7.107263400030206e-08, 1.6379799205878953e-07, 6.876715730186334e-08, 3.4624252975845593e-07, 2.0256192101442139e-07, 6.19600768914097e-07, 2.6840686118134727e-08, 6.720200644849683e-07, 1.1348156476742588e-06, 1.8397651047052932e-06, 6.397185075002199e-07, 7.257429501805746e-08, 4.2213261508550204e-07, 3.965735402289283e-07, 1.4037237860975438e-07, 3.249813573802385e-07, 1.5857477819736232e-07, 1.1122037335553614e-07, 7.39131564841955e-08, 3.4231769063808315e-07, 5.39788445053091e-08, 8.517191218970765e-08, 4.060975243191933e-06, 1.4478493540082127e-05, 7.316928396505773e-09, 6.948383646232514e-09, 4.4688626132938225e-08, 9.23126250995665e-08, 5.41190452452156e-08, 2.224259958438779e-07, 1.7609368185844687e-08, 2.0905980946395175e-08, 3.6797160873902612e-09, 6.177843658861093e-08, 1.7920085326750268e-07, 2.6278852871541858e-08, 2.698776313536655e-08, 1.6432605320915172e-07, 1.2827440798446332e-07, 4.468845560268164e-08, 6.316474099321567e-08, 1.9461522171582146e-08, 2.0871000927513705e-08, 2.2414262446091016e-08, 2.4764982953229264e-08, 6.785314532464781e-09, 2.4248494767675766e-08, 4.554914667664889e-09, 2.8976625543464252e-08, 2.0402152856036082e-08, 1.6009290959573264e-07, 2.019951637066697e-07, 1.6111656009343278e-08, 5.964040639128143e-08, 4.0982603977113286e-09, 3.907972612182675e-08, 7.511245847524606e-09, 5.965555942566425e-07, 1.6478051634294388e-07, 1.410681527147517e-08, 3.285524030616216e-08, 3.338656018314623e-09, 1.2200259469352659e-08, 4.624581961820695e-08, 6.842250765970448e-09, 1.7426071252657493e-08, 4.661075436729334e-08, 9.33183343931887e-08, 1.230591379908219e-07, 1.2359244294657401e-08, 1.1173082903326303e-08, 2.772451956900568e-08, 2.4191731995415466e-07, 3.4511506896706123e-07, 2.593734649281032e-08, 9.964443847820803e-08, 9.797671118860762e-09, 1.90853270964908e-07, 3.9726497647052383e-08, 2.6638923955601967e-08, 6.8740502179309715e-09, 3.146955052102385e-08, 2.4086276084744895e-07, 1.3126738451774145e-07, 2.1254035686979478e-07, 2.0501781605730685e-08, 3.694923123020999e-08, 6.563101351275691e-07, 2.5600135344916453e-08, 2.688264899575188e-08, 6.880517844365386e-07, 2.0092501529234141e-07, 2.7880052044793047e-08, 2.62837644982028e-08, 5.167780159354152e-08, 1.893516099471526e-07, 4.6184757707123936e-07, 1.1086663143089481e-08, 1.4542384008109366e-07, 2.8736709012378014e-08, 6.105074703555147e-07, 1.2016337258557996e-08, 1.3944533350240818e-07, 2.0936788303060894e-08, 4.3800788773751265e-08, 1.9662968142597492e-08, 6.6543535126584175e-09, 1.1149432488366529e-08, 6.424846787922434e-08, 6.971495380980741e-09, 3.2599853927450795e-09, 1.4260012726197147e-08, 2.7894840215481054e-08, 8.115682703646598e-08, 2.599534454361674e-08, 2.285521105704902e-08, 1.05594978094814e-07, 8.145473628928812e-08, 3.779314994289962e-08, 4.8818307618603285e-08, 2.3420332340151617e-08, 1.0599223720930695e-08, 3.604067799756194e-08, 5.062377184117395e-08, 3.6804056691153164e-08, 1.5015616527946918e-09, 1.4474869658442913e-06, 1.0761959856608883e-06, 1.304976251503831e-07, 3.073563448197092e-08, 1.7183838352252678e-08, 2.042087920983704e-08, 7.904924714807748e-09, 1.6901843480354728e-07, 1.2416376371504612e-08, 5.4757610001843204e-08, 2.6250097207025647e-08, 1.3261202802539174e-08, 6.298007804161898e-08, 1.2709827856838274e-08, 2.0171410142211244e-07, 4.386575014336813e-08, 6.962276444255622e-08, 2.956475668725034e-07, 7.925040677037032e-07, 2.0868660044470744e-07, 1.7341581326490996e-07, 4.294189182019181e-08, 4.213362547744737e-09, 8.8246935092684e-08, 1.7341355018629656e-08, 7.321497008661026e-08, 1.7941481189609476e-08, 1.1245008835203407e-07, 4.2423405943736725e-07, 8.259470440918903e-09, 1.1336252470073305e-07, 8.268681739309613e-08, 2.2186704029536486e-08, 1.95395184476 [e2e-predictor] 86532e-08, 1.0675572426066537e-08, 3.288476690954667e-08, 2.4340661042288048e-08, 6.639029948019015e-08, 5.604618547039308e-09, 1.386590220420203e-08, 6.6757905869963e-08, 1.1355724360839758e-08, 3.1321272331297223e-07, 3.1244848486267074e-08, 1.502163371469578e-07, 1.346161226933873e-08, 1.8882738572756352e-07, 4.645693962856967e-08, 4.645321638463429e-08, 7.71435715307689e-09, 3.585664032357272e-08, 7.608256247237932e-09, 4.2214487905312126e-08, 4.3406713778892936e-09, 1.3401408871516196e-08, 8.565495335233209e-08, 1.7045202582721686e-08, 5.422122839604526e-08, 3.0218750168842234e-08, 6.153300091682468e-08, 3.9388163841636015e-09, 4.135585029985123e-08, 1.7818981845607595e-08, 4.31053557292671e-08, 3.9032919119108556e-09, 7.663005163749403e-08, 1.1890258555524724e-08, 9.304103265606045e-09, 1.0967905472725192e-09, 1.0536636452229686e-08, 1.1516662112853737e-07, 8.134422273542441e-07, 5.9525618212319387e-08, 2.8063100288022724e-08, 1.283296402476708e-08, 1.060557863752365e-07, 7.872866945035639e-07, 2.7501052102252288e-08, 3.936232229051484e-09, 2.0519189902756807e-08, 7.394696410756296e-09, 3.5983390489491285e-08, 2.5378296797384792e-08, 4.6989228508209635e-08, 7.549436631393291e-09, 6.322734407149255e-07, 5.581916262542563e-09, 1.2963950268840563e-07, 1.587476106124086e-08, 3.3837391555380236e-08, 6.474394353972457e-09, 9.121036015358186e-08, 1.3918366015275296e-08, 8.229938508463874e-09, 2.7060956142577197e-08, 2.6095642979839795e-08, 5.722282203635132e-09, 6.963402938708896e-07, 4.685033871965061e-08, 9.590480409826796e-09, 2.0991818416860042e-07, 3.082116961650172e-08, 3.563118511351604e-08, 7.326215154535021e-07, 2.1759417450084584e-06, 2.4074907400972734e-07, 2.974484800688515e-07, 2.528977205429328e-08, 7.667869894589785e-09, 2.6632636718204594e-07, 3.435845030708151e-08, 2.3129958393042216e-08, 3.1239306252928145e-08, 2.8691238185274415e-07, 3.8957892911639647e-08, 2.4183783864373254e-08, 1.158230222841894e-08, 5.1544706280992614e-08, 2.0343204454320585e-08, 8.201909196259294e-08, 1.1641392738681589e-08, 5.496298740581551e-07, 1.1682007361457636e-08, 4.7576321549058775e-08, 1.6349654075042963e-08, 4.090812311119407e-08, 2.1270925287808495e-07, 1.669703095785735e-07, 3.989651631286506e-08, 2.8524043500510743e-06, 1.2500227342115977e-07, 2.4846258384059183e-07, 1.2454135500661323e-08, 2.9699904757762852e-08, 4.25093871214699e-09, 1.6144196024470148e-07, 2.6385819751340023e-07, 7.637975585339518e-09, 3.4455236441743864e-09, 7.273212787595185e-08, 1.763124757303558e-08, 7.586505645917896e-09, 2.154678924171094e-08, 1.267521554382256e-07, 2.5636829548147944e-08, 3.500932521660616e-08, 6.472151170555662e-08, 8.387794458997178e-09, 3.0694742747527926e-08, 7.520281286588215e-08, 1.5724816648798878e-07, 1.9633705505839316e-07, 1.2290654183289007e-07, 1.1121071841202479e-09, 1.5468767600168576e-08, 9.916888643601851e-09, 6.882375487293757e-07, 8.267530660077682e-08, 4.46946799570469e-08, 2.075171856574798e-08, 8.649272587035739e-08, 5.20271186132959e-08, 4.5564465978031876e-08, 2.031970502969216e-08, 8.705073994974555e-09, 6.45198596771479e-08, 2.1777212921847422e-08, 1.0309374687267336e-08, 3.2118705917127954e-08, 2.3336649235261575e-07, 8.053995692591798e-09, 1.9623111313649133e-07, 1.2887916511772346e-07, 1.5392334518082862e-08, 1.401880500573327e-09, 5.696758975659577e-08, 6.079961423210989e-09, 1.0782660986308201e-08, 2.4260382147645032e-08, 1.9388348704296732e-08, 2.2969983604070876e-07, 1.997151066746028e-08, 2.8477696645268225e-08, 5.227270349905666e-08, 2.739241153904004e-07, 9.85714976309282e-08, 3.129077086327925e-08, 4.1513992243835673e-08, 5.2511412995670526e-09, 1.5806674582563573e-06, 8.547505672140687e-07, 1.0688979479311911e-08, 1.0621677120070672e-06, 7.737217089243131e-08, 6.394137130882882e-08, 1.1698201518584028e-07, 1.0486479595783749e-07, 2.1160776952910965e-07, 1.5339432835048683e-08, 5.094390331805698e-08, 1.4005206772083056e-08, 2.6281709253339613e-08, 8.77830927947798e-08, 7.771985721660712e-09, 4.228822803042931e-08, 3.3243432540075446e-07, 7.72913466562386e-08, 7.63679175452836e-10, 5.9894155 [e2e-predictor] 41307608e-08, 1.3260753917165857e-07, 1.2853475084284582e-07, 8.844149412823299e-09, 1.0194228394766469e-07, 2.4937534703894926e-07, 1.6547782877296413e-07, 1.176258734858493e-08, 1.1496064900029523e-07, 2.934237670615403e-07, 1.3261050924029405e-08, 8.630173198298507e-08, 5.739413211358624e-08, 1.1093964502606468e-07, 2.2933429022486962e-07, 3.470581191322708e-07, 1.475092403779854e-07, 1.5024763655446804e-08, 6.454251177956394e-08, 5.164459437878577e-08, 6.23732674398525e-08, 1.2935890936205396e-07, 1.4051898311606692e-08, 5.386868906498421e-08, 2.082729722019394e-08, 1.3040525104202061e-08, 1.05788906523685e-07, 1.507953939494655e-08, 8.926216423787992e-07, 4.637372086335745e-08, 7.480928161385236e-07, 5.883074472023964e-09, 2.870727655590599e-09, 8.381493898923509e-07, 7.341868002441743e-09, 1.4245849833116608e-08, 1.0926260785026898e-07, 1.1308060265946551e-07, 2.52335809136639e-07, 1.178269073420779e-07, 4.667730024721095e-08, 2.79587997198405e-09, 3.436349871321909e-08, 1.4674341741738317e-07, 3.539640758276619e-08, 2.0580911552769976e-07, 7.18380377406902e-08, 2.7693602078215918e-08, 4.5492825506698864e-08, 1.955898198957584e-08, 1.5359576721607482e-08, 1.2336422550163206e-08, 2.957029643368969e-08, 2.877517033539334e-07, 9.051751135302766e-07, 2.3732353326977318e-07, 1.6521470413977113e-08, 1.5478654802336678e-08, 3.526755776306345e-08, 3.61636587342673e-08, 1.615889466677345e-08, 7.649993705172164e-08, 1.9661243300106435e-08, 4.9174737171142624e-08, 1.1712456426948847e-07, 1.0889120716228717e-08, 1.4941074368834961e-06, 1.0185750909386115e-08, 3.7575578204496196e-08, 2.0970718495050278e-08, 3.368522527580353e-08, 4.845519985252622e-09, 6.039555842107802e-07, 1.0373183201295433e-08, 2.841620698745828e-07, 4.499007388858445e-07, 3.4631430168019506e-08, 7.720588968140873e-08, 1.4711066853578814e-07, 1.160143270340086e-07, 4.0074306184578745e-07, 3.0256124716743216e-08, 6.706726196625823e-08, 2.012845534693497e-08, 1.5987545731732666e-09, 4.1919221871467016e-08, 1.3166965295852151e-08, 3.231780709711529e-08, 9.247563781400459e-08, 1.3075164417841734e-07, 1.0574149911235509e-07, 3.762125899697821e-08, 1.0942090256094161e-07, 7.001387558602801e-08, 2.7426720450307585e-08, 2.0766329100752046e-08, 4.5402845927355884e-08, 3.390360703292572e-08, 1.0469531730450399e-07, 2.8271228913467894e-08, 3.406190955956845e-07, 5.146158059687878e-07, 6.740644380442973e-07, 6.3821934404018066e-09, 3.638748680145909e-08, 3.6260146885069844e-08, 1.6065344254911906e-07, 3.639007957190188e-07, 6.232625970881145e-09, 4.8054303647404595e-08, 3.3725854109434295e-08, 6.3287848206528e-07, 6.480550496235082e-08, 2.116489525860743e-07, 8.38768769995113e-08, 1.758889212055692e-08, 2.729004577872729e-09, 2.144768451728396e-08, 7.861175532752895e-08, 2.011793931444572e-08, 2.840733515085958e-08, 2.492219834948628e-07, 2.0156347702027233e-08, 2.6551370879701608e-08, 2.7847898209643063e-08, 6.907038496706264e-09, 1.8805204504701578e-08, 1.3006692611838844e-08, 3.685865976876812e-07, 3.9678877783444477e-07, 2.7591790185965692e-08, 2.5228539257682314e-08, 1.5469838388071366e-07, 3.6892611632310945e-08, 1.4401622117077295e-09, 2.1504664715621402e-08, 5.068058328561165e-08, 5.081657761252245e-08, 1.1718628734058711e-08, 5.409200909411993e-08, 7.138175419640902e-07, 2.523685509459028e-07, 7.071956531490287e-08, 7.199660245760242e-08, 1.2525339876390262e-08, 3.4802988579940575e-07, 1.9591639954796847e-07, 1.2404500182583433e-07, 1.2346009725661133e-07, 1.92011366806355e-08, 1.9895020386684337e-07, 3.7876212388709973e-08, 1.0629652713589621e-08, 1.2436973229057458e-08, 2.186162078032794e-07, 2.618108396745811e-07, 1.1128912547064829e-07, 1.0776497560982534e-07, 6.380246109216614e-09, 3.895029632161595e-09, 1.576254788915321e-07, 2.9089968212048234e-09, 1.03809227880447e-08, 2.8134863327977655e-08, 1.0777869263733919e-08, 1.3605805371241786e-08, 2.9236161935841665e-08, 1.918957508451058e-07, 2.1994748067299952e-07, 1.3263833587018325e-08, 4.900423533626963e-08, 2.9808058243929736e-09, 8.925934658066126e-09, 1.0996840771326788e-08, 7.715520666806697e-09, 7.45 [e2e-predictor] 4385553273823e-09, 5.0860990086221136e-08, 1.5129604946650943e-07, 1.1959995660504319e-08, 1.1323217030678734e-08, 9.39119804144184e-09, 9.585582461113518e-08, 1.9055129030220996e-08, 1.8105044574667772e-08, 6.179151057494892e-08, 6.389322493305372e-08, 1.1853056314237165e-08, 9.372677745034252e-09, 1.2332006349424773e-07, 1.6521848777983905e-08, 1.246099685658919e-07, 4.196111902388111e-09, 3.9965367903960214e-08, 1.255440107428285e-08, 1.4302015571843185e-08, 6.631698745707126e-09, 5.964162053118116e-09, 5.556878956980427e-09, 1.419230528654225e-07, 1.7612896030527736e-08, 3.3801444487835397e-07, 7.856438344333583e-08, 2.9667463152804885e-08, 2.8991748877160717e-06, 1.3787196166958893e-06, 5.313577133847502e-09, 2.5128217728820346e-08, 8.406522056247923e-08, 4.492775573794461e-08, 5.4087468726038423e-08, 2.4238921980668238e-08, 4.0167356019082945e-07, 4.1083040969169815e-08, 5.415281023601892e-08, 8.640659743264223e-09, 5.773174294176897e-08, 2.6442751277500065e-07, 8.953106203080097e-07, 2.799365539374321e-08, 5.889395637836969e-09, 4.178830081968954e-08, 2.888028660663622e-08, 2.1357223545237503e-08, 2.3023956430279213e-07, 8.786517469161481e-08, 2.069703963769598e-09, 2.2363829899063603e-08, 3.2032425600903025e-09, 1.176860120466472e-08, 6.963498577761129e-08, 2.2719042647167953e-08, 7.360319465021803e-09, 6.922469264480924e-09, 3.2138252947788715e-08, 1.3705631829452614e-07, 1.9814805796158907e-08, 1.0578844467090676e-08, 2.7049035011827982e-08, 2.9755513608620277e-09, 2.1773475467057324e-08, 1.0975404585167325e-08, 1.9918440230526357e-08, 2.3881801425318372e-08, 2.1357100976615584e-08, 6.109046157831699e-09, 1.1890372242362446e-08, 1.1459750481890296e-08, 3.731681275098708e-08, 1.5726065782928345e-08, 3.403987491878979e-08, 3.6921193213856895e-08, 9.281668411631472e-08, 1.2323060616381554e-07, 4.234707162709128e-08, 1.7423577247654976e-08, 5.811278214196136e-08, 3.9313878374969136e-08, 2.3690166273127033e-08, 1.792834680713895e-08, 1.4406491999352511e-08, 7.0197305745978156e-09, 6.041436506620812e-08, 4.867428771149207e-08, 1.0685186957459791e-08, 1.005112082452797e-08, 4.2425732260653604e-08, 2.6077879411445792e-08, 5.136626679558276e-09, 1.6972753202892932e-09, 1.9131350725842822e-08, 2.1117190840413969e-07, 1.3372530638378066e-08, 2.000219723186092e-08, 1.0454146348592985e-07, 2.814382682458927e-08, 2.134452614654947e-07, 2.104584950757271e-08, 1.9114216343041335e-07, 3.957645233754192e-08, 2.9315843974586642e-08, 1.1051893977764848e-07, 4.840025980001883e-08, 5.583537188158516e-08, 1.2129960680340446e-07, 1.7761975001917563e-08, 2.561053769056798e-08, 1.2038481145282276e-07, 4.6748018789344314e-09, 1.591587306393194e-08, 3.147561500327356e-08, 6.147622144681009e-08, 2.204614979461894e-08, 3.2577264619249036e-07, 1.198899752807847e-07, 2.3818718375423487e-07, 1.490982981522393e-08, 5.1016343149967724e-08, 5.5141523347401744e-08, 2.288526701477167e-08, 5.7140397302646306e-08, 5.185041231925425e-07, 4.977233558633998e-08, 1.104976110610778e-08, 1.26408238543263e-07, 8.174780674607973e-08]]} [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_paddle.py::test_paddle [e2e-predictor] predictor/test_paddle.py::test_paddle_runtime 2026-07-28 17:22:14.996 4494 kserve.trace Host from isvc status URL = isvc-paddle-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:22:14.996 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-paddle-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:22:14.996 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:22:14.996 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:22:14.997 4494 kserve.trace Using external route host: isvc-paddle-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:22:14.997 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-paddle-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:22:15.036 4494 kserve.trace Sending Header = {'Host': 'isvc-paddle-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:22:15.036 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-paddle-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:22:15.036 4494 kserve.trace base url = https://isvc-paddle-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:22:15.036 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-paddle-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:22:16.551 4494 kserve.trace Got response {'predictions': [[6.736569968524009e-09, 1.153582651625129e-08, 5.142186410012073e-08, 6.647101002954514e-08, 4.094442118685038e-08, 1.3402272713847196e-07, 9.35544619551365e-08, 2.8935589924117266e-08, 6.845294819868286e-08, 7.680520752728626e-08, 2.0334418877610005e-06, 1.1085519417974865e-06, 2.3477346644540376e-07, 6.581955744877632e-07, 0.00012372950732242316, 4.2878312456196e-07, 6.419879809982376e-06, 0.9993487000465393, 7.371911488007754e-05, 3.101079710177146e-05, 5.602713827101979e-06, 2.1862256289750803e-06, 1.9543842100233633e-08, 3.728848128048412e-07, 4.29031416615544e-07, 1.8250955235998845e-07, 7.159837878134567e-08, 9.231503561579757e-09, 6.469168170042394e-07, 7.031603299623157e-09, 4.4511686780879245e-08, 1.2455829789814743e-07, 9.446210924579646e-08, 4.3476589439706004e-08, 4.6581672563661414e-07, 6.797636586952649e-08, 2.1060014887552825e-07, 2.2604888627597575e-08, 1.4311326879123953e-07, 7.951216218771151e-08, 1.2341666888460168e-07, 1.0921577313638409e-06, 1.5243762391037308e-05, 3.1172930903267115e-07, 2.415180802017858e-07, 6.863702850523623e-08, 8.46760883632669e-08, 9.424560687421035e-08, 1.0219084245477461e-08, 3.3770398033539095e-08, 3.6928447855189006e-08, 1.3693848188722768e-07, 1.067415098532365e-07, 2.5994486918534676e-07, 3.486601087843155e-07, 3.132020509610811e-08, 3.5748260529544496e-07, 6.648343742199359e-08, 3.163856945320731e-07, 1.2095728152416996e-06, 8.663983663836916e-08, 4.014367860349921e-08, 1.2544606420306081e-07, 3.320077013313494e-08, 1.973121896980956e-07, 3.8063546980993124e-07, 1.382770449254167e-07, 2.3001977211833946e-08, 7.14414980507172e-08, 2.851079372589993e-08, 2.982524804906461e-08, 8.935905526641363e-08, 6.223806394700659e-07, 6.478746428228987e-08, 1.3662854314588913e-07, 9.973038572752557e-08, 2.576123669939534e-08, 4.130169273253159e-08, 3.9383902361578293e-08, 1.2157941853274679e-07, 4.302776233089389e-06, 1.8178942582380841e-06, 1.8520014464229462e-06, 1.6245859342234326e-06, 1.64481571118813e-05, 1.0544756150920875e-05, 3.993004611402284e-06, 2.646454788646224e-07, 1.919327405630611e-05, 4.803187607649306e-07, 1.6962640358997305e-07, 4.550436642603017e-06, 4.2358526116004214e-05, 4.443309080670588e-06, 5.103951934870565e-06, 1.3506267350749113e-05, 4.1758286783988297e-07, 4.4944314936401497e-07, 3.156656873670727e-07, 1.0557428140600678e-06, 1.3364497952750298e-08, 1.3893487249561076e-08, 6.770295613023336e-08, 1.4129548731034447e-07, 7.170429228153807e-08, 7.934354329108828e-08, 2.6391164809069778e-08, 2.6133946562367782e-08, 7.1966232972897615e-09, 2.1752097012495142e-08, 6.684530973188885e-08, 3.4177467966856057e-08, 1.6228092647452286e-07, 4.107051836399478e-07, 6.472068889706861e-07, 2.9513313393181306e-07, 5.653383094994524e-09, 4.8300755395302986e-08, 8.887371727439586e-09, 3.7305639466467255e-08, 1.7784008932153483e-08, 4.641856232723285e-09, 3.413062898971475e-08, 1.9373679549516964e-07, 1.2980041219634586e-06, 3.5640496776068176e-08, 2.1493017143825455e-08, 3.055253330330743e-07, 1.553226809392072e-07, 1.4520828699460253e-06, 3.488421285169352e-08, 3.825387466349639e-05, 4.508809183789708e-07, 4.176648928932991e-07, 6.770537765987683e-07, 1.4142071336209483e-07, 1.423571211489616e-05, 6.293736873885791e-07, 4.762812295666663e-06, 9.024806217894366e-07, 9.058892942448438e-07, 1.5713549146312289e-06, 1.5720408441666223e-07, 1.8185124872616143e-07, 7.193072093514274e-08, 1.1952631666645175e-06, 8.874753234522359e-07, 2.0870552930318809e-07, 9.906116105184992e-08, 7.793509837483725e-09, 1.0058372623689138e-07, 4.205883783470199e-07, 1.8436001880672848e-07, 1.6437729755125474e-07, 7.02525184692604e-08, 2.5704193262754416e-07, 7.586770323086967e-08, 7.841231308702845e-07, 2.495280568837188e-07, 5.157617977147311e-08, 4.0673629797538524e-08, 7.531674839356128e-09, 4.797425745550754e-08, 1.7419722553313477e-08, 1.7958475950763386e-07, 1.2566235163546935e-08, 8.975329990335013e-08, 3.269607518063822e-08, 1.1208221195602164e-07, 3.906690437815996e-08, 4.676846998563633e-08, 1.852334321483795e-07, 1. [e2e-predictor] 4832883721282997e-07, 1.2279221550670627e-07, 1.0729002042353386e-06, 3.6537970427019673e-09, 1.6198204377815273e-07, 1.6190487173162182e-08, 1.2004819893718377e-07, 1.480009448329156e-08, 4.022906807676918e-08, 2.1505718450498534e-07, 1.192555032503151e-07, 4.898190297808469e-08, 7.608841201545147e-08, 2.3137237903370078e-08, 8.52094430570105e-08, 9.586112525994395e-08, 1.3351485961266008e-07, 3.0216618540634954e-08, 4.4238134933038964e-08, 2.610634552979718e-08, 2.397684113475407e-07, 1.3192399705985736e-07, 1.6734691854480843e-08, 1.5883161097463017e-07, 4.064300753725547e-07, 8.753363545110915e-08, 8.366871497855755e-07, 3.437562767771851e-08, 7.847781802183817e-08, 8.526320982582547e-09, 9.601245665180613e-08, 5.258879127723048e-07, 1.3557291822507977e-07, 1.0307118003538562e-07, 1.0429684671464656e-08, 5.187630236491714e-08, 2.1869702493404475e-08, 1.179131725592697e-08, 2.9806198398318884e-08, 4.3383401759911067e-08, 2.999067660880428e-08, 2.8507258775789523e-08, 3.058627484620047e-08, 6.441018740588333e-08, 1.5363882610586188e-08, 1.597368459727022e-08, 2.57365329048298e-08, 1.090363141997841e-07, 3.211827603877282e-08, 6.819658615597746e-09, 1.9251054084179486e-07, 5.825749838095362e-08, 1.876556723345857e-07, 4.0070369777822634e-07, 1.5791380647556252e-08, 1.9501307235714194e-07, 1.0141919304373914e-08, 2.744775962071344e-08, 1.2843395680306458e-08, 3.7296960186949946e-08, 7.407404467585366e-08, 4.206033921150265e-08, 1.6924593282396927e-08, 1.4591853414458456e-07, 4.344931525679385e-08, 1.7191209167322086e-07, 3.5816999144344663e-08, 8.440128596021168e-09, 4.1947775031303536e-08, 2.513996122388562e-08, 2.8339847801817086e-08, 8.747072399728495e-08, 8.277008411994302e-09, 1.1676148936601294e-08, 1.4548319882123906e-07, 7.200192886358536e-09, 2.623576165206032e-06, 5.675644842995098e-07, 1.9483265987219056e-06, 6.752511438889996e-08, 8.168359499904909e-08, 1.0932910754490877e-07, 1.6708914074570203e-07, 3.138676518688044e-08, 2.9734879447573803e-08, 5.7520821883372264e-08, 5.8507939826313304e-08, 3.2544284067625995e-07, 3.330167075432655e-08, 4.186742614820105e-07, 1.5085691984495497e-07, 2.334644193524582e-07, 2.8639897209359333e-07, 2.2940034227758588e-07, 1.8537357959758083e-07, 3.151756686747831e-07, 1.1075776455982123e-06, 1.5369606387594104e-07, 1.9237216974943294e-07, 1.6404264613356645e-07, 2.900799245253438e-07, 1.2466382770526252e-07, 5.802550617772795e-08, 5.1861558603150115e-08, 6.009345909774311e-09, 1.23330778478703e-07, 1.3798263864828186e-07, 1.7370017246776115e-07, 5.617696388071636e-07, 5.160393001801822e-08, 4.813227150179955e-08, 8.032599652096906e-08, 2.0644911273848265e-06, 5.638538027596951e-07, 8.794099244369136e-07, 3.4785582556651207e-06, 2.972355162000895e-07, 3.390414349269122e-07, 9.468957529179534e-08, 3.754799493549399e-08, 1.5678830322940485e-07, 8.2030041426151e-08, 6.847877820348458e-09, 1.825139683830912e-08, 6.050154155445853e-08, 3.9562941367421445e-08, 1.0699794472657231e-07, 3.256623699599004e-07, 3.536899839673424e-07, 7.326217854597417e-08, 4.857605517827324e-07, 7.717640073678922e-07, 3.456741737295488e-08, 3.2461647947457095e-07, 3.1608240078639938e-06, 5.330938535053065e-08, 3.645648973815696e-07, 5.481521156980307e-07, 4.623012372917401e-08, 1.346600981833035e-07, 4.352903104631878e-08, 1.6403919289587066e-07, 2.4636646500653114e-08, 5.958662185889807e-08, 9.493569308460792e-08, 5.5234050222452424e-08, 5.741187081298449e-08, 1.1850225746457e-05, 5.826300366607029e-06, 7.420818292303011e-06, 9.127862199420633e-07, 2.001935399675858e-06, 1.0334889566365746e-06, 3.514648483360361e-08, 2.0589775431290036e-06, 3.5655068586493144e-07, 6.873136015883574e-08, 2.193502490754895e-09, 5.560284321859399e-08, 3.3266616128457827e-07, 1.3073517379780242e-07, 2.7187292417352182e-08, 1.0462799515664756e-08, 7.466601914529747e-07, 6.923080064780152e-08, 1.6145493830777013e-08, 8.568415310605815e-09, 4.762142236103273e-09, 1.233958357715892e-07, 8.340525603500737e-09, 3.2648699299642203e-09, 5.063196084620358e-09, 4.0704488668552585e-09, 1.2043389396865223e-08, 5.1055555339019065e-09, 7 [e2e-predictor] .267066948202228e-09, 1.1844983305309142e-07, 7.53547197973603e-08, 6.386873252495207e-08, 1.621267564644313e-08, 2.6103893446816073e-07, 6.979319664424111e-07, 6.647392325476176e-08, 7.717375183347031e-07, 6.651137596236367e-07, 3.3244478458982485e-07, 3.707229154770175e-07, 3.995585302618565e-07, 6.411539743567118e-08, 7.107263400030206e-08, 1.6379799205878953e-07, 6.876715730186334e-08, 3.4624252975845593e-07, 2.0256192101442139e-07, 6.19600768914097e-07, 2.6840686118134727e-08, 6.720200644849683e-07, 1.1348156476742588e-06, 1.8397651047052932e-06, 6.397185075002199e-07, 7.257429501805746e-08, 4.2213261508550204e-07, 3.965735402289283e-07, 1.4037237860975438e-07, 3.249813573802385e-07, 1.5857477819736232e-07, 1.1122037335553614e-07, 7.39131564841955e-08, 3.4231769063808315e-07, 5.39788445053091e-08, 8.517191218970765e-08, 4.060975243191933e-06, 1.4478493540082127e-05, 7.316928396505773e-09, 6.948383646232514e-09, 4.4688626132938225e-08, 9.23126250995665e-08, 5.41190452452156e-08, 2.224259958438779e-07, 1.7609368185844687e-08, 2.0905980946395175e-08, 3.6797160873902612e-09, 6.177843658861093e-08, 1.7920085326750268e-07, 2.6278852871541858e-08, 2.698776313536655e-08, 1.6432605320915172e-07, 1.2827440798446332e-07, 4.468845560268164e-08, 6.316474099321567e-08, 1.9461522171582146e-08, 2.0871000927513705e-08, 2.2414262446091016e-08, 2.4764982953229264e-08, 6.785314532464781e-09, 2.4248494767675766e-08, 4.554914667664889e-09, 2.8976625543464252e-08, 2.0402152856036082e-08, 1.6009290959573264e-07, 2.019951637066697e-07, 1.6111656009343278e-08, 5.964040639128143e-08, 4.0982603977113286e-09, 3.907972612182675e-08, 7.511245847524606e-09, 5.965555942566425e-07, 1.6478051634294388e-07, 1.410681527147517e-08, 3.285524030616216e-08, 3.338656018314623e-09, 1.2200259469352659e-08, 4.624581961820695e-08, 6.842250765970448e-09, 1.7426071252657493e-08, 4.661075436729334e-08, 9.33183343931887e-08, 1.230591379908219e-07, 1.2359244294657401e-08, 1.1173082903326303e-08, 2.772451956900568e-08, 2.4191731995415466e-07, 3.4511506896706123e-07, 2.593734649281032e-08, 9.964443847820803e-08, 9.797671118860762e-09, 1.90853270964908e-07, 3.9726497647052383e-08, 2.6638923955601967e-08, 6.8740502179309715e-09, 3.146955052102385e-08, 2.4086276084744895e-07, 1.3126738451774145e-07, 2.1254035686979478e-07, 2.0501781605730685e-08, 3.694923123020999e-08, 6.563101351275691e-07, 2.5600135344916453e-08, 2.688264899575188e-08, 6.880517844365386e-07, 2.0092501529234141e-07, 2.7880052044793047e-08, 2.62837644982028e-08, 5.167780159354152e-08, 1.893516099471526e-07, 4.6184757707123936e-07, 1.1086663143089481e-08, 1.4542384008109366e-07, 2.8736709012378014e-08, 6.105074703555147e-07, 1.2016337258557996e-08, 1.3944533350240818e-07, 2.0936788303060894e-08, 4.3800788773751265e-08, 1.9662968142597492e-08, 6.6543535126584175e-09, 1.1149432488366529e-08, 6.424846787922434e-08, 6.971495380980741e-09, 3.2599853927450795e-09, 1.4260012726197147e-08, 2.7894840215481054e-08, 8.115682703646598e-08, 2.599534454361674e-08, 2.285521105704902e-08, 1.05594978094814e-07, 8.145473628928812e-08, 3.779314994289962e-08, 4.8818307618603285e-08, 2.3420332340151617e-08, 1.0599223720930695e-08, 3.604067799756194e-08, 5.062377184117395e-08, 3.6804056691153164e-08, 1.5015616527946918e-09, 1.4474869658442913e-06, 1.0761959856608883e-06, 1.304976251503831e-07, 3.073563448197092e-08, 1.7183838352252678e-08, 2.042087920983704e-08, 7.904924714807748e-09, 1.6901843480354728e-07, 1.2416376371504612e-08, 5.4757610001843204e-08, 2.6250097207025647e-08, 1.3261202802539174e-08, 6.298007804161898e-08, 1.2709827856838274e-08, 2.0171410142211244e-07, 4.386575014336813e-08, 6.962276444255622e-08, 2.956475668725034e-07, 7.925040677037032e-07, 2.0868660044470744e-07, 1.7341581326490996e-07, 4.294189182019181e-08, 4.213362547744737e-09, 8.8246935092684e-08, 1.7341355018629656e-08, 7.321497008661026e-08, 1.7941481189609476e-08, 1.1245008835203407e-07, 4.2423405943736725e-07, 8.259470440918903e-09, 1.1336252470073305e-07, 8.268681739309613e-08, 2.2186704029536486e-08, 1.9539518447686532e-08, 1.0675572426066537e [e2e-predictor] -08, 3.288476690954667e-08, 2.4340661042288048e-08, 6.639029948019015e-08, 5.604618547039308e-09, 1.386590220420203e-08, 6.6757905869963e-08, 1.1355724360839758e-08, 3.1321272331297223e-07, 3.1244848486267074e-08, 1.502163371469578e-07, 1.346161226933873e-08, 1.8882738572756352e-07, 4.645693962856967e-08, 4.645321638463429e-08, 7.71435715307689e-09, 3.585664032357272e-08, 7.608256247237932e-09, 4.2214487905312126e-08, 4.3406713778892936e-09, 1.3401408871516196e-08, 8.565495335233209e-08, 1.7045202582721686e-08, 5.422122839604526e-08, 3.0218750168842234e-08, 6.153300091682468e-08, 3.9388163841636015e-09, 4.135585029985123e-08, 1.7818981845607595e-08, 4.31053557292671e-08, 3.9032919119108556e-09, 7.663005163749403e-08, 1.1890258555524724e-08, 9.304103265606045e-09, 1.0967905472725192e-09, 1.0536636452229686e-08, 1.1516662112853737e-07, 8.134422273542441e-07, 5.9525618212319387e-08, 2.8063100288022724e-08, 1.283296402476708e-08, 1.060557863752365e-07, 7.872866945035639e-07, 2.7501052102252288e-08, 3.936232229051484e-09, 2.0519189902756807e-08, 7.394696410756296e-09, 3.5983390489491285e-08, 2.5378296797384792e-08, 4.6989228508209635e-08, 7.549436631393291e-09, 6.322734407149255e-07, 5.581916262542563e-09, 1.2963950268840563e-07, 1.587476106124086e-08, 3.3837391555380236e-08, 6.474394353972457e-09, 9.121036015358186e-08, 1.3918366015275296e-08, 8.229938508463874e-09, 2.7060956142577197e-08, 2.6095642979839795e-08, 5.722282203635132e-09, 6.963402938708896e-07, 4.685033871965061e-08, 9.590480409826796e-09, 2.0991818416860042e-07, 3.082116961650172e-08, 3.563118511351604e-08, 7.326215154535021e-07, 2.1759417450084584e-06, 2.4074907400972734e-07, 2.974484800688515e-07, 2.528977205429328e-08, 7.667869894589785e-09, 2.6632636718204594e-07, 3.435845030708151e-08, 2.3129958393042216e-08, 3.1239306252928145e-08, 2.8691238185274415e-07, 3.8957892911639647e-08, 2.4183783864373254e-08, 1.158230222841894e-08, 5.1544706280992614e-08, 2.0343204454320585e-08, 8.201909196259294e-08, 1.1641392738681589e-08, 5.496298740581551e-07, 1.1682007361457636e-08, 4.7576321549058775e-08, 1.6349654075042963e-08, 4.090812311119407e-08, 2.1270925287808495e-07, 1.669703095785735e-07, 3.989651631286506e-08, 2.8524043500510743e-06, 1.2500227342115977e-07, 2.4846258384059183e-07, 1.2454135500661323e-08, 2.9699904757762852e-08, 4.25093871214699e-09, 1.6144196024470148e-07, 2.6385819751340023e-07, 7.637975585339518e-09, 3.4455236441743864e-09, 7.273212787595185e-08, 1.763124757303558e-08, 7.586505645917896e-09, 2.154678924171094e-08, 1.267521554382256e-07, 2.5636829548147944e-08, 3.500932521660616e-08, 6.472151170555662e-08, 8.387794458997178e-09, 3.0694742747527926e-08, 7.520281286588215e-08, 1.5724816648798878e-07, 1.9633705505839316e-07, 1.2290654183289007e-07, 1.1121071841202479e-09, 1.5468767600168576e-08, 9.916888643601851e-09, 6.882375487293757e-07, 8.267530660077682e-08, 4.46946799570469e-08, 2.075171856574798e-08, 8.649272587035739e-08, 5.20271186132959e-08, 4.5564465978031876e-08, 2.031970502969216e-08, 8.705073994974555e-09, 6.45198596771479e-08, 2.1777212921847422e-08, 1.0309374687267336e-08, 3.2118705917127954e-08, 2.3336649235261575e-07, 8.053995692591798e-09, 1.9623111313649133e-07, 1.2887916511772346e-07, 1.5392334518082862e-08, 1.401880500573327e-09, 5.696758975659577e-08, 6.079961423210989e-09, 1.0782660986308201e-08, 2.4260382147645032e-08, 1.9388348704296732e-08, 2.2969983604070876e-07, 1.997151066746028e-08, 2.8477696645268225e-08, 5.227270349905666e-08, 2.739241153904004e-07, 9.85714976309282e-08, 3.129077086327925e-08, 4.1513992243835673e-08, 5.2511412995670526e-09, 1.5806674582563573e-06, 8.547505672140687e-07, 1.0688979479311911e-08, 1.0621677120070672e-06, 7.737217089243131e-08, 6.394137130882882e-08, 1.1698201518584028e-07, 1.0486479595783749e-07, 2.1160776952910965e-07, 1.5339432835048683e-08, 5.094390331805698e-08, 1.4005206772083056e-08, 2.6281709253339613e-08, 8.77830927947798e-08, 7.771985721660712e-09, 4.228822803042931e-08, 3.3243432540075446e-07, 7.72913466562386e-08, 7.63679175452836e-10, 5.989415541307608e-08, 1.32607539171658 [e2e-predictor] 57e-07, 1.2853475084284582e-07, 8.844149412823299e-09, 1.0194228394766469e-07, 2.4937534703894926e-07, 1.6547782877296413e-07, 1.176258734858493e-08, 1.1496064900029523e-07, 2.934237670615403e-07, 1.3261050924029405e-08, 8.630173198298507e-08, 5.739413211358624e-08, 1.1093964502606468e-07, 2.2933429022486962e-07, 3.470581191322708e-07, 1.475092403779854e-07, 1.5024763655446804e-08, 6.454251177956394e-08, 5.164459437878577e-08, 6.23732674398525e-08, 1.2935890936205396e-07, 1.4051898311606692e-08, 5.386868906498421e-08, 2.082729722019394e-08, 1.3040525104202061e-08, 1.05788906523685e-07, 1.507953939494655e-08, 8.926216423787992e-07, 4.637372086335745e-08, 7.480928161385236e-07, 5.883074472023964e-09, 2.870727655590599e-09, 8.381493898923509e-07, 7.341868002441743e-09, 1.4245849833116608e-08, 1.0926260785026898e-07, 1.1308060265946551e-07, 2.52335809136639e-07, 1.178269073420779e-07, 4.667730024721095e-08, 2.79587997198405e-09, 3.436349871321909e-08, 1.4674341741738317e-07, 3.539640758276619e-08, 2.0580911552769976e-07, 7.18380377406902e-08, 2.7693602078215918e-08, 4.5492825506698864e-08, 1.955898198957584e-08, 1.5359576721607482e-08, 1.2336422550163206e-08, 2.957029643368969e-08, 2.877517033539334e-07, 9.051751135302766e-07, 2.3732353326977318e-07, 1.6521470413977113e-08, 1.5478654802336678e-08, 3.526755776306345e-08, 3.61636587342673e-08, 1.615889466677345e-08, 7.649993705172164e-08, 1.9661243300106435e-08, 4.9174737171142624e-08, 1.1712456426948847e-07, 1.0889120716228717e-08, 1.4941074368834961e-06, 1.0185750909386115e-08, 3.7575578204496196e-08, 2.0970718495050278e-08, 3.368522527580353e-08, 4.845519985252622e-09, 6.039555842107802e-07, 1.0373183201295433e-08, 2.841620698745828e-07, 4.499007388858445e-07, 3.4631430168019506e-08, 7.720588968140873e-08, 1.4711066853578814e-07, 1.160143270340086e-07, 4.0074306184578745e-07, 3.0256124716743216e-08, 6.706726196625823e-08, 2.012845534693497e-08, 1.5987545731732666e-09, 4.1919221871467016e-08, 1.3166965295852151e-08, 3.231780709711529e-08, 9.247563781400459e-08, 1.3075164417841734e-07, 1.0574149911235509e-07, 3.762125899697821e-08, 1.0942090256094161e-07, 7.001387558602801e-08, 2.7426720450307585e-08, 2.0766329100752046e-08, 4.5402845927355884e-08, 3.390360703292572e-08, 1.0469531730450399e-07, 2.8271228913467894e-08, 3.406190955956845e-07, 5.146158059687878e-07, 6.740644380442973e-07, 6.3821934404018066e-09, 3.638748680145909e-08, 3.6260146885069844e-08, 1.6065344254911906e-07, 3.639007957190188e-07, 6.232625970881145e-09, 4.8054303647404595e-08, 3.3725854109434295e-08, 6.3287848206528e-07, 6.480550496235082e-08, 2.116489525860743e-07, 8.38768769995113e-08, 1.758889212055692e-08, 2.729004577872729e-09, 2.144768451728396e-08, 7.861175532752895e-08, 2.011793931444572e-08, 2.840733515085958e-08, 2.492219834948628e-07, 2.0156347702027233e-08, 2.6551370879701608e-08, 2.7847898209643063e-08, 6.907038496706264e-09, 1.8805204504701578e-08, 1.3006692611838844e-08, 3.685865976876812e-07, 3.9678877783444477e-07, 2.7591790185965692e-08, 2.5228539257682314e-08, 1.5469838388071366e-07, 3.6892611632310945e-08, 1.4401622117077295e-09, 2.1504664715621402e-08, 5.068058328561165e-08, 5.081657761252245e-08, 1.1718628734058711e-08, 5.409200909411993e-08, 7.138175419640902e-07, 2.523685509459028e-07, 7.071956531490287e-08, 7.199660245760242e-08, 1.2525339876390262e-08, 3.4802988579940575e-07, 1.9591639954796847e-07, 1.2404500182583433e-07, 1.2346009725661133e-07, 1.92011366806355e-08, 1.9895020386684337e-07, 3.7876212388709973e-08, 1.0629652713589621e-08, 1.2436973229057458e-08, 2.186162078032794e-07, 2.618108396745811e-07, 1.1128912547064829e-07, 1.0776497560982534e-07, 6.380246109216614e-09, 3.895029632161595e-09, 1.576254788915321e-07, 2.9089968212048234e-09, 1.03809227880447e-08, 2.8134863327977655e-08, 1.0777869263733919e-08, 1.3605805371241786e-08, 2.9236161935841665e-08, 1.918957508451058e-07, 2.1994748067299952e-07, 1.3263833587018325e-08, 4.900423533626963e-08, 2.9808058243929736e-09, 8.925934658066126e-09, 1.0996840771326788e-08, 7.715520666806697e-09, 7.454385553273823e-09, 5.086099008 [e2e-predictor] 6221136e-08, 1.5129604946650943e-07, 1.1959995660504319e-08, 1.1323217030678734e-08, 9.39119804144184e-09, 9.585582461113518e-08, 1.9055129030220996e-08, 1.8105044574667772e-08, 6.179151057494892e-08, 6.389322493305372e-08, 1.1853056314237165e-08, 9.372677745034252e-09, 1.2332006349424773e-07, 1.6521848777983905e-08, 1.246099685658919e-07, 4.196111902388111e-09, 3.9965367903960214e-08, 1.255440107428285e-08, 1.4302015571843185e-08, 6.631698745707126e-09, 5.964162053118116e-09, 5.556878956980427e-09, 1.419230528654225e-07, 1.7612896030527736e-08, 3.3801444487835397e-07, 7.856438344333583e-08, 2.9667463152804885e-08, 2.8991748877160717e-06, 1.3787196166958893e-06, 5.313577133847502e-09, 2.5128217728820346e-08, 8.406522056247923e-08, 4.492775573794461e-08, 5.4087468726038423e-08, 2.4238921980668238e-08, 4.0167356019082945e-07, 4.1083040969169815e-08, 5.415281023601892e-08, 8.640659743264223e-09, 5.773174294176897e-08, 2.6442751277500065e-07, 8.953106203080097e-07, 2.799365539374321e-08, 5.889395637836969e-09, 4.178830081968954e-08, 2.888028660663622e-08, 2.1357223545237503e-08, 2.3023956430279213e-07, 8.786517469161481e-08, 2.069703963769598e-09, 2.2363829899063603e-08, 3.2032425600903025e-09, 1.176860120466472e-08, 6.963498577761129e-08, 2.2719042647167953e-08, 7.360319465021803e-09, 6.922469264480924e-09, 3.2138252947788715e-08, 1.3705631829452614e-07, 1.9814805796158907e-08, 1.0578844467090676e-08, 2.7049035011827982e-08, 2.9755513608620277e-09, 2.1773475467057324e-08, 1.0975404585167325e-08, 1.9918440230526357e-08, 2.3881801425318372e-08, 2.1357100976615584e-08, 6.109046157831699e-09, 1.1890372242362446e-08, 1.1459750481890296e-08, 3.731681275098708e-08, 1.5726065782928345e-08, 3.403987491878979e-08, 3.6921193213856895e-08, 9.281668411631472e-08, 1.2323060616381554e-07, 4.234707162709128e-08, 1.7423577247654976e-08, 5.811278214196136e-08, 3.9313878374969136e-08, 2.3690166273127033e-08, 1.792834680713895e-08, 1.4406491999352511e-08, 7.0197305745978156e-09, 6.041436506620812e-08, 4.867428771149207e-08, 1.0685186957459791e-08, 1.005112082452797e-08, 4.2425732260653604e-08, 2.6077879411445792e-08, 5.136626679558276e-09, 1.6972753202892932e-09, 1.9131350725842822e-08, 2.1117190840413969e-07, 1.3372530638378066e-08, 2.000219723186092e-08, 1.0454146348592985e-07, 2.814382682458927e-08, 2.134452614654947e-07, 2.104584950757271e-08, 1.9114216343041335e-07, 3.957645233754192e-08, 2.9315843974586642e-08, 1.1051893977764848e-07, 4.840025980001883e-08, 5.583537188158516e-08, 1.2129960680340446e-07, 1.7761975001917563e-08, 2.561053769056798e-08, 1.2038481145282276e-07, 4.6748018789344314e-09, 1.591587306393194e-08, 3.147561500327356e-08, 6.147622144681009e-08, 2.204614979461894e-08, 3.2577264619249036e-07, 1.198899752807847e-07, 2.3818718375423487e-07, 1.490982981522393e-08, 5.1016343149967724e-08, 5.5141523347401744e-08, 2.288526701477167e-08, 5.7140397302646306e-08, 5.185041231925425e-07, 4.977233558633998e-08, 1.104976110610778e-08, 1.26408238543263e-07, 8.174780674607973e-08]]} [e2e-predictor] 2026-07-28 17:22:16.551 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [[6.736569968524009e-09, 1.153582651625129e-08, 5.142186410012073e-08, 6.647101002954514e-08, 4.094442118685038e-08, 1.3402272713847196e-07, 9.35544619551365e-08, 2.8935589924117266e-08, 6.845294819868286e-08, 7.680520752728626e-08, 2.0334418877610005e-06, 1.1085519417974865e-06, 2.3477346644540376e-07, 6.581955744877632e-07, 0.00012372950732242316, 4.2878312456196e-07, 6.419879809982376e-06, 0.9993487000465393, 7.371911488007754e-05, 3.101079710177146e-05, 5.602713827101979e-06, 2.1862256289750803e-06, 1.9543842100233633e-08, 3.728848128048412e-07, 4.29031416615544e-07, 1.8250955235998845e-07, 7.159837878134567e-08, 9.231503561579757e-09, 6.469168170042394e-07, 7.031603299623157e-09, 4.4511686780879245e-08, 1.2455829789814743e-07, 9.446210924579646e-08, 4.3476589439706004e-08, 4.6581672563661414e-07, 6.797636586952649e-08, 2.1060014887552825e-07, 2.2604888627597575e-08, 1.4311326879123953e-07, 7.951216218771151e-08, 1.2341666888460168e-07, 1.0921577313638409e-06, 1.5243762391037308e-05, 3.1172930903267115e-07, 2.415180802017858e-07, 6.863702850523623e-08, 8.46760883632669e-08, 9.424560687421035e-08, 1.0219084245477461e-08, 3.3770398033539095e-08, 3.6928447855189006e-08, 1.3693848188722768e-07, 1.067415098532365e-07, 2.5994486918534676e-07, 3.486601087843155e-07, 3.132020509610811e-08, 3.5748260529544496e-07, 6.648343742199359e-08, 3.163856945320731e-07, 1.2095728152416996e-06, 8.663983663836916e-08, 4.014367860349921e-08, 1.2544606420306081e-07, 3.320077013313494e-08, 1.973121896980956e-07, 3.8063546980993124e-07, 1.382770449254167e-07, 2.3001977211833946e-08, 7.14414980507172e-08, 2.851079372589993e-08, 2.982524804906461e-08, 8.935905526641363e-08, 6.223806394700659e-07, 6.478746428228987e-08, 1.3662854314588913e-07, 9.973038572752557e-08, 2.576123669939534e-08, 4.130169273253159e-08, 3.9383902361578293e-08, 1.2157941853274679e-07, 4.302776233089389e-06, 1.8178942582380841e-06, 1.8520014464229462e-06, 1.6245859342234326e-06, 1.64481571118813e-05, 1.0544756150920875e-05, 3.993004611402284e-06, 2.646454788646224e-07, 1.919327405630611e-05, 4.803187607649306e-07, 1.6962640358997305e-07, 4.550436642603017e-06, 4.2358526116004214e-05, 4.443309080670588e-06, 5.103951934870565e-06, 1.3506267350749113e-05, 4.1758286783988297e-07, 4.4944314936401497e-07, 3.156656873670727e-07, 1.0557428140600678e-06, 1.3364497952750298e-08, 1.3893487249561076e-08, 6.770295613023336e-08, 1.4129548731034447e-07, 7.170429228153807e-08, 7.934354329108828e-08, 2.6391164809069778e-08, 2.6133946562367782e-08, 7.1966232972897615e-09, 2.1752097012495142e-08, 6.684530973188885e-08, 3.4177467966856057e-08, 1.6228092647452286e-07, 4.107051836399478e-07, 6.472068889706861e-07, 2.9513313393181306e-07, 5.653383094994524e-09, 4.8300755395302986e-08, 8.887371727439586e-09, 3.7305639466467255e-08, 1.7784008932153483e-08, 4.641856232723285e-09, 3.413062898971475e-08, 1.9373679549516964e-07, 1.2980041219634586e-06, 3.5640496776068176e-08, 2.1493017143825455e-08, 3.055253330330743e-07, 1.553226809392072e-07, 1.4520828699460253e-06, 3.488421285169352e-08, 3.825387466349639e-05, 4.508809183789708e-07, 4.176648928932991e-07, 6.770537765987683e-07, 1.4142071336209483e-07, 1.423571211489616e-05, 6.293736873885791e-07, 4.762812295666663e-06, 9.024806217894366e-07, 9.058892942448438e-07, 1.5713549146312289e-06, 1.5720408441666223e-07, 1.8185124872616143e-07, 7.193072093514274e-08, 1.1952631666645175e-06, 8.874753234522359e-07, 2.0870552930318809e-07, 9.906116105184992e-08, 7.793509837483725e-09, 1.0058372623689138e-07, 4.205883783470199e-07, 1.8436001880672848e-07, 1.6437729755125474e-07, 7.02525184692604e-08, 2.5704193262754416e-07, 7.586770323086967e-08, 7.841231308702845e-07, 2.495280568837188e-07, 5.157617977147311e-08, 4.0673629797538524e-08, 7.531674839356128e-09, 4.797425745550754e-08, 1.7419722553313477e-08, 1.7958475950763386e-07, 1.2566235163546935e-08, 8.975329990335013e-08, 3.269607518063822e-08, 1.1208221195602164e-07, 3.906690437815996e-08, 4.676846998563633e [e2e-predictor] -08, 1.852334321483795e-07, 1.4832883721282997e-07, 1.2279221550670627e-07, 1.0729002042353386e-06, 3.6537970427019673e-09, 1.6198204377815273e-07, 1.6190487173162182e-08, 1.2004819893718377e-07, 1.480009448329156e-08, 4.022906807676918e-08, 2.1505718450498534e-07, 1.192555032503151e-07, 4.898190297808469e-08, 7.608841201545147e-08, 2.3137237903370078e-08, 8.52094430570105e-08, 9.586112525994395e-08, 1.3351485961266008e-07, 3.0216618540634954e-08, 4.4238134933038964e-08, 2.610634552979718e-08, 2.397684113475407e-07, 1.3192399705985736e-07, 1.6734691854480843e-08, 1.5883161097463017e-07, 4.064300753725547e-07, 8.753363545110915e-08, 8.366871497855755e-07, 3.437562767771851e-08, 7.847781802183817e-08, 8.526320982582547e-09, 9.601245665180613e-08, 5.258879127723048e-07, 1.3557291822507977e-07, 1.0307118003538562e-07, 1.0429684671464656e-08, 5.187630236491714e-08, 2.1869702493404475e-08, 1.179131725592697e-08, 2.9806198398318884e-08, 4.3383401759911067e-08, 2.999067660880428e-08, 2.8507258775789523e-08, 3.058627484620047e-08, 6.441018740588333e-08, 1.5363882610586188e-08, 1.597368459727022e-08, 2.57365329048298e-08, 1.090363141997841e-07, 3.211827603877282e-08, 6.819658615597746e-09, 1.9251054084179486e-07, 5.825749838095362e-08, 1.876556723345857e-07, 4.0070369777822634e-07, 1.5791380647556252e-08, 1.9501307235714194e-07, 1.0141919304373914e-08, 2.744775962071344e-08, 1.2843395680306458e-08, 3.7296960186949946e-08, 7.407404467585366e-08, 4.206033921150265e-08, 1.6924593282396927e-08, 1.4591853414458456e-07, 4.344931525679385e-08, 1.7191209167322086e-07, 3.5816999144344663e-08, 8.440128596021168e-09, 4.1947775031303536e-08, 2.513996122388562e-08, 2.8339847801817086e-08, 8.747072399728495e-08, 8.277008411994302e-09, 1.1676148936601294e-08, 1.4548319882123906e-07, 7.200192886358536e-09, 2.623576165206032e-06, 5.675644842995098e-07, 1.9483265987219056e-06, 6.752511438889996e-08, 8.168359499904909e-08, 1.0932910754490877e-07, 1.6708914074570203e-07, 3.138676518688044e-08, 2.9734879447573803e-08, 5.7520821883372264e-08, 5.8507939826313304e-08, 3.2544284067625995e-07, 3.330167075432655e-08, 4.186742614820105e-07, 1.5085691984495497e-07, 2.334644193524582e-07, 2.8639897209359333e-07, 2.2940034227758588e-07, 1.8537357959758083e-07, 3.151756686747831e-07, 1.1075776455982123e-06, 1.5369606387594104e-07, 1.9237216974943294e-07, 1.6404264613356645e-07, 2.900799245253438e-07, 1.2466382770526252e-07, 5.802550617772795e-08, 5.1861558603150115e-08, 6.009345909774311e-09, 1.23330778478703e-07, 1.3798263864828186e-07, 1.7370017246776115e-07, 5.617696388071636e-07, 5.160393001801822e-08, 4.813227150179955e-08, 8.032599652096906e-08, 2.0644911273848265e-06, 5.638538027596951e-07, 8.794099244369136e-07, 3.4785582556651207e-06, 2.972355162000895e-07, 3.390414349269122e-07, 9.468957529179534e-08, 3.754799493549399e-08, 1.5678830322940485e-07, 8.2030041426151e-08, 6.847877820348458e-09, 1.825139683830912e-08, 6.050154155445853e-08, 3.9562941367421445e-08, 1.0699794472657231e-07, 3.256623699599004e-07, 3.536899839673424e-07, 7.326217854597417e-08, 4.857605517827324e-07, 7.717640073678922e-07, 3.456741737295488e-08, 3.2461647947457095e-07, 3.1608240078639938e-06, 5.330938535053065e-08, 3.645648973815696e-07, 5.481521156980307e-07, 4.623012372917401e-08, 1.346600981833035e-07, 4.352903104631878e-08, 1.6403919289587066e-07, 2.4636646500653114e-08, 5.958662185889807e-08, 9.493569308460792e-08, 5.5234050222452424e-08, 5.741187081298449e-08, 1.1850225746457e-05, 5.826300366607029e-06, 7.420818292303011e-06, 9.127862199420633e-07, 2.001935399675858e-06, 1.0334889566365746e-06, 3.514648483360361e-08, 2.0589775431290036e-06, 3.5655068586493144e-07, 6.873136015883574e-08, 2.193502490754895e-09, 5.560284321859399e-08, 3.3266616128457827e-07, 1.3073517379780242e-07, 2.7187292417352182e-08, 1.0462799515664756e-08, 7.466601914529747e-07, 6.923080064780152e-08, 1.6145493830777013e-08, 8.568415310605815e-09, 4.762142236103273e-09, 1.233958357715892e-07, 8.340525603500737e-09, 3.2648699299642203e-09, 5.063196084620358e-09, 4.0704488668552585e-09, 1.2043389396865223e [e2e-predictor] -08, 5.1055555339019065e-09, 7.267066948202228e-09, 1.1844983305309142e-07, 7.53547197973603e-08, 6.386873252495207e-08, 1.621267564644313e-08, 2.6103893446816073e-07, 6.979319664424111e-07, 6.647392325476176e-08, 7.717375183347031e-07, 6.651137596236367e-07, 3.3244478458982485e-07, 3.707229154770175e-07, 3.995585302618565e-07, 6.411539743567118e-08, 7.107263400030206e-08, 1.6379799205878953e-07, 6.876715730186334e-08, 3.4624252975845593e-07, 2.0256192101442139e-07, 6.19600768914097e-07, 2.6840686118134727e-08, 6.720200644849683e-07, 1.1348156476742588e-06, 1.8397651047052932e-06, 6.397185075002199e-07, 7.257429501805746e-08, 4.2213261508550204e-07, 3.965735402289283e-07, 1.4037237860975438e-07, 3.249813573802385e-07, 1.5857477819736232e-07, 1.1122037335553614e-07, 7.39131564841955e-08, 3.4231769063808315e-07, 5.39788445053091e-08, 8.517191218970765e-08, 4.060975243191933e-06, 1.4478493540082127e-05, 7.316928396505773e-09, 6.948383646232514e-09, 4.4688626132938225e-08, 9.23126250995665e-08, 5.41190452452156e-08, 2.224259958438779e-07, 1.7609368185844687e-08, 2.0905980946395175e-08, 3.6797160873902612e-09, 6.177843658861093e-08, 1.7920085326750268e-07, 2.6278852871541858e-08, 2.698776313536655e-08, 1.6432605320915172e-07, 1.2827440798446332e-07, 4.468845560268164e-08, 6.316474099321567e-08, 1.9461522171582146e-08, 2.0871000927513705e-08, 2.2414262446091016e-08, 2.4764982953229264e-08, 6.785314532464781e-09, 2.4248494767675766e-08, 4.554914667664889e-09, 2.8976625543464252e-08, 2.0402152856036082e-08, 1.6009290959573264e-07, 2.019951637066697e-07, 1.6111656009343278e-08, 5.964040639128143e-08, 4.0982603977113286e-09, 3.907972612182675e-08, 7.511245847524606e-09, 5.965555942566425e-07, 1.6478051634294388e-07, 1.410681527147517e-08, 3.285524030616216e-08, 3.338656018314623e-09, 1.2200259469352659e-08, 4.624581961820695e-08, 6.842250765970448e-09, 1.7426071252657493e-08, 4.661075436729334e-08, 9.33183343931887e-08, 1.230591379908219e-07, 1.2359244294657401e-08, 1.1173082903326303e-08, 2.772451956900568e-08, 2.4191731995415466e-07, 3.4511506896706123e-07, 2.593734649281032e-08, 9.964443847820803e-08, 9.797671118860762e-09, 1.90853270964908e-07, 3.9726497647052383e-08, 2.6638923955601967e-08, 6.8740502179309715e-09, 3.146955052102385e-08, 2.4086276084744895e-07, 1.3126738451774145e-07, 2.1254035686979478e-07, 2.0501781605730685e-08, 3.694923123020999e-08, 6.563101351275691e-07, 2.5600135344916453e-08, 2.688264899575188e-08, 6.880517844365386e-07, 2.0092501529234141e-07, 2.7880052044793047e-08, 2.62837644982028e-08, 5.167780159354152e-08, 1.893516099471526e-07, 4.6184757707123936e-07, 1.1086663143089481e-08, 1.4542384008109366e-07, 2.8736709012378014e-08, 6.105074703555147e-07, 1.2016337258557996e-08, 1.3944533350240818e-07, 2.0936788303060894e-08, 4.3800788773751265e-08, 1.9662968142597492e-08, 6.6543535126584175e-09, 1.1149432488366529e-08, 6.424846787922434e-08, 6.971495380980741e-09, 3.2599853927450795e-09, 1.4260012726197147e-08, 2.7894840215481054e-08, 8.115682703646598e-08, 2.599534454361674e-08, 2.285521105704902e-08, 1.05594978094814e-07, 8.145473628928812e-08, 3.779314994289962e-08, 4.8818307618603285e-08, 2.3420332340151617e-08, 1.0599223720930695e-08, 3.604067799756194e-08, 5.062377184117395e-08, 3.6804056691153164e-08, 1.5015616527946918e-09, 1.4474869658442913e-06, 1.0761959856608883e-06, 1.304976251503831e-07, 3.073563448197092e-08, 1.7183838352252678e-08, 2.042087920983704e-08, 7.904924714807748e-09, 1.6901843480354728e-07, 1.2416376371504612e-08, 5.4757610001843204e-08, 2.6250097207025647e-08, 1.3261202802539174e-08, 6.298007804161898e-08, 1.2709827856838274e-08, 2.0171410142211244e-07, 4.386575014336813e-08, 6.962276444255622e-08, 2.956475668725034e-07, 7.925040677037032e-07, 2.0868660044470744e-07, 1.7341581326490996e-07, 4.294189182019181e-08, 4.213362547744737e-09, 8.8246935092684e-08, 1.7341355018629656e-08, 7.321497008661026e-08, 1.7941481189609476e-08, 1.1245008835203407e-07, 4.2423405943736725e-07, 8.259470440918903e-09, 1.1336252470073305e-07, 8.268681739309613e-08, 2.2186704029536486e-08, 1.95395184476 [e2e-predictor] 86532e-08, 1.0675572426066537e-08, 3.288476690954667e-08, 2.4340661042288048e-08, 6.639029948019015e-08, 5.604618547039308e-09, 1.386590220420203e-08, 6.6757905869963e-08, 1.1355724360839758e-08, 3.1321272331297223e-07, 3.1244848486267074e-08, 1.502163371469578e-07, 1.346161226933873e-08, 1.8882738572756352e-07, 4.645693962856967e-08, 4.645321638463429e-08, 7.71435715307689e-09, 3.585664032357272e-08, 7.608256247237932e-09, 4.2214487905312126e-08, 4.3406713778892936e-09, 1.3401408871516196e-08, 8.565495335233209e-08, 1.7045202582721686e-08, 5.422122839604526e-08, 3.0218750168842234e-08, 6.153300091682468e-08, 3.9388163841636015e-09, 4.135585029985123e-08, 1.7818981845607595e-08, 4.31053557292671e-08, 3.9032919119108556e-09, 7.663005163749403e-08, 1.1890258555524724e-08, 9.304103265606045e-09, 1.0967905472725192e-09, 1.0536636452229686e-08, 1.1516662112853737e-07, 8.134422273542441e-07, 5.9525618212319387e-08, 2.8063100288022724e-08, 1.283296402476708e-08, 1.060557863752365e-07, 7.872866945035639e-07, 2.7501052102252288e-08, 3.936232229051484e-09, 2.0519189902756807e-08, 7.394696410756296e-09, 3.5983390489491285e-08, 2.5378296797384792e-08, 4.6989228508209635e-08, 7.549436631393291e-09, 6.322734407149255e-07, 5.581916262542563e-09, 1.2963950268840563e-07, 1.587476106124086e-08, 3.3837391555380236e-08, 6.474394353972457e-09, 9.121036015358186e-08, 1.3918366015275296e-08, 8.229938508463874e-09, 2.7060956142577197e-08, 2.6095642979839795e-08, 5.722282203635132e-09, 6.963402938708896e-07, 4.685033871965061e-08, 9.590480409826796e-09, 2.0991818416860042e-07, 3.082116961650172e-08, 3.563118511351604e-08, 7.326215154535021e-07, 2.1759417450084584e-06, 2.4074907400972734e-07, 2.974484800688515e-07, 2.528977205429328e-08, 7.667869894589785e-09, 2.6632636718204594e-07, 3.435845030708151e-08, 2.3129958393042216e-08, 3.1239306252928145e-08, 2.8691238185274415e-07, 3.8957892911639647e-08, 2.4183783864373254e-08, 1.158230222841894e-08, 5.1544706280992614e-08, 2.0343204454320585e-08, 8.201909196259294e-08, 1.1641392738681589e-08, 5.496298740581551e-07, 1.1682007361457636e-08, 4.7576321549058775e-08, 1.6349654075042963e-08, 4.090812311119407e-08, 2.1270925287808495e-07, 1.669703095785735e-07, 3.989651631286506e-08, 2.8524043500510743e-06, 1.2500227342115977e-07, 2.4846258384059183e-07, 1.2454135500661323e-08, 2.9699904757762852e-08, 4.25093871214699e-09, 1.6144196024470148e-07, 2.6385819751340023e-07, 7.637975585339518e-09, 3.4455236441743864e-09, 7.273212787595185e-08, 1.763124757303558e-08, 7.586505645917896e-09, 2.154678924171094e-08, 1.267521554382256e-07, 2.5636829548147944e-08, 3.500932521660616e-08, 6.472151170555662e-08, 8.387794458997178e-09, 3.0694742747527926e-08, 7.520281286588215e-08, 1.5724816648798878e-07, 1.9633705505839316e-07, 1.2290654183289007e-07, 1.1121071841202479e-09, 1.5468767600168576e-08, 9.916888643601851e-09, 6.882375487293757e-07, 8.267530660077682e-08, 4.46946799570469e-08, 2.075171856574798e-08, 8.649272587035739e-08, 5.20271186132959e-08, 4.5564465978031876e-08, 2.031970502969216e-08, 8.705073994974555e-09, 6.45198596771479e-08, 2.1777212921847422e-08, 1.0309374687267336e-08, 3.2118705917127954e-08, 2.3336649235261575e-07, 8.053995692591798e-09, 1.9623111313649133e-07, 1.2887916511772346e-07, 1.5392334518082862e-08, 1.401880500573327e-09, 5.696758975659577e-08, 6.079961423210989e-09, 1.0782660986308201e-08, 2.4260382147645032e-08, 1.9388348704296732e-08, 2.2969983604070876e-07, 1.997151066746028e-08, 2.8477696645268225e-08, 5.227270349905666e-08, 2.739241153904004e-07, 9.85714976309282e-08, 3.129077086327925e-08, 4.1513992243835673e-08, 5.2511412995670526e-09, 1.5806674582563573e-06, 8.547505672140687e-07, 1.0688979479311911e-08, 1.0621677120070672e-06, 7.737217089243131e-08, 6.394137130882882e-08, 1.1698201518584028e-07, 1.0486479595783749e-07, 2.1160776952910965e-07, 1.5339432835048683e-08, 5.094390331805698e-08, 1.4005206772083056e-08, 2.6281709253339613e-08, 8.77830927947798e-08, 7.771985721660712e-09, 4.228822803042931e-08, 3.3243432540075446e-07, 7.72913466562386e-08, 7.63679175452836e-10, 5.9894155 [e2e-predictor] 41307608e-08, 1.3260753917165857e-07, 1.2853475084284582e-07, 8.844149412823299e-09, 1.0194228394766469e-07, 2.4937534703894926e-07, 1.6547782877296413e-07, 1.176258734858493e-08, 1.1496064900029523e-07, 2.934237670615403e-07, 1.3261050924029405e-08, 8.630173198298507e-08, 5.739413211358624e-08, 1.1093964502606468e-07, 2.2933429022486962e-07, 3.470581191322708e-07, 1.475092403779854e-07, 1.5024763655446804e-08, 6.454251177956394e-08, 5.164459437878577e-08, 6.23732674398525e-08, 1.2935890936205396e-07, 1.4051898311606692e-08, 5.386868906498421e-08, 2.082729722019394e-08, 1.3040525104202061e-08, 1.05788906523685e-07, 1.507953939494655e-08, 8.926216423787992e-07, 4.637372086335745e-08, 7.480928161385236e-07, 5.883074472023964e-09, 2.870727655590599e-09, 8.381493898923509e-07, 7.341868002441743e-09, 1.4245849833116608e-08, 1.0926260785026898e-07, 1.1308060265946551e-07, 2.52335809136639e-07, 1.178269073420779e-07, 4.667730024721095e-08, 2.79587997198405e-09, 3.436349871321909e-08, 1.4674341741738317e-07, 3.539640758276619e-08, 2.0580911552769976e-07, 7.18380377406902e-08, 2.7693602078215918e-08, 4.5492825506698864e-08, 1.955898198957584e-08, 1.5359576721607482e-08, 1.2336422550163206e-08, 2.957029643368969e-08, 2.877517033539334e-07, 9.051751135302766e-07, 2.3732353326977318e-07, 1.6521470413977113e-08, 1.5478654802336678e-08, 3.526755776306345e-08, 3.61636587342673e-08, 1.615889466677345e-08, 7.649993705172164e-08, 1.9661243300106435e-08, 4.9174737171142624e-08, 1.1712456426948847e-07, 1.0889120716228717e-08, 1.4941074368834961e-06, 1.0185750909386115e-08, 3.7575578204496196e-08, 2.0970718495050278e-08, 3.368522527580353e-08, 4.845519985252622e-09, 6.039555842107802e-07, 1.0373183201295433e-08, 2.841620698745828e-07, 4.499007388858445e-07, 3.4631430168019506e-08, 7.720588968140873e-08, 1.4711066853578814e-07, 1.160143270340086e-07, 4.0074306184578745e-07, 3.0256124716743216e-08, 6.706726196625823e-08, 2.012845534693497e-08, 1.5987545731732666e-09, 4.1919221871467016e-08, 1.3166965295852151e-08, 3.231780709711529e-08, 9.247563781400459e-08, 1.3075164417841734e-07, 1.0574149911235509e-07, 3.762125899697821e-08, 1.0942090256094161e-07, 7.001387558602801e-08, 2.7426720450307585e-08, 2.0766329100752046e-08, 4.5402845927355884e-08, 3.390360703292572e-08, 1.0469531730450399e-07, 2.8271228913467894e-08, 3.406190955956845e-07, 5.146158059687878e-07, 6.740644380442973e-07, 6.3821934404018066e-09, 3.638748680145909e-08, 3.6260146885069844e-08, 1.6065344254911906e-07, 3.639007957190188e-07, 6.232625970881145e-09, 4.8054303647404595e-08, 3.3725854109434295e-08, 6.3287848206528e-07, 6.480550496235082e-08, 2.116489525860743e-07, 8.38768769995113e-08, 1.758889212055692e-08, 2.729004577872729e-09, 2.144768451728396e-08, 7.861175532752895e-08, 2.011793931444572e-08, 2.840733515085958e-08, 2.492219834948628e-07, 2.0156347702027233e-08, 2.6551370879701608e-08, 2.7847898209643063e-08, 6.907038496706264e-09, 1.8805204504701578e-08, 1.3006692611838844e-08, 3.685865976876812e-07, 3.9678877783444477e-07, 2.7591790185965692e-08, 2.5228539257682314e-08, 1.5469838388071366e-07, 3.6892611632310945e-08, 1.4401622117077295e-09, 2.1504664715621402e-08, 5.068058328561165e-08, 5.081657761252245e-08, 1.1718628734058711e-08, 5.409200909411993e-08, 7.138175419640902e-07, 2.523685509459028e-07, 7.071956531490287e-08, 7.199660245760242e-08, 1.2525339876390262e-08, 3.4802988579940575e-07, 1.9591639954796847e-07, 1.2404500182583433e-07, 1.2346009725661133e-07, 1.92011366806355e-08, 1.9895020386684337e-07, 3.7876212388709973e-08, 1.0629652713589621e-08, 1.2436973229057458e-08, 2.186162078032794e-07, 2.618108396745811e-07, 1.1128912547064829e-07, 1.0776497560982534e-07, 6.380246109216614e-09, 3.895029632161595e-09, 1.576254788915321e-07, 2.9089968212048234e-09, 1.03809227880447e-08, 2.8134863327977655e-08, 1.0777869263733919e-08, 1.3605805371241786e-08, 2.9236161935841665e-08, 1.918957508451058e-07, 2.1994748067299952e-07, 1.3263833587018325e-08, 4.900423533626963e-08, 2.9808058243929736e-09, 8.925934658066126e-09, 1.0996840771326788e-08, 7.715520666806697e-09, 7.45 [e2e-predictor] 4385553273823e-09, 5.0860990086221136e-08, 1.5129604946650943e-07, 1.1959995660504319e-08, 1.1323217030678734e-08, 9.39119804144184e-09, 9.585582461113518e-08, 1.9055129030220996e-08, 1.8105044574667772e-08, 6.179151057494892e-08, 6.389322493305372e-08, 1.1853056314237165e-08, 9.372677745034252e-09, 1.2332006349424773e-07, 1.6521848777983905e-08, 1.246099685658919e-07, 4.196111902388111e-09, 3.9965367903960214e-08, 1.255440107428285e-08, 1.4302015571843185e-08, 6.631698745707126e-09, 5.964162053118116e-09, 5.556878956980427e-09, 1.419230528654225e-07, 1.7612896030527736e-08, 3.3801444487835397e-07, 7.856438344333583e-08, 2.9667463152804885e-08, 2.8991748877160717e-06, 1.3787196166958893e-06, 5.313577133847502e-09, 2.5128217728820346e-08, 8.406522056247923e-08, 4.492775573794461e-08, 5.4087468726038423e-08, 2.4238921980668238e-08, 4.0167356019082945e-07, 4.1083040969169815e-08, 5.415281023601892e-08, 8.640659743264223e-09, 5.773174294176897e-08, 2.6442751277500065e-07, 8.953106203080097e-07, 2.799365539374321e-08, 5.889395637836969e-09, 4.178830081968954e-08, 2.888028660663622e-08, 2.1357223545237503e-08, 2.3023956430279213e-07, 8.786517469161481e-08, 2.069703963769598e-09, 2.2363829899063603e-08, 3.2032425600903025e-09, 1.176860120466472e-08, 6.963498577761129e-08, 2.2719042647167953e-08, 7.360319465021803e-09, 6.922469264480924e-09, 3.2138252947788715e-08, 1.3705631829452614e-07, 1.9814805796158907e-08, 1.0578844467090676e-08, 2.7049035011827982e-08, 2.9755513608620277e-09, 2.1773475467057324e-08, 1.0975404585167325e-08, 1.9918440230526357e-08, 2.3881801425318372e-08, 2.1357100976615584e-08, 6.109046157831699e-09, 1.1890372242362446e-08, 1.1459750481890296e-08, 3.731681275098708e-08, 1.5726065782928345e-08, 3.403987491878979e-08, 3.6921193213856895e-08, 9.281668411631472e-08, 1.2323060616381554e-07, 4.234707162709128e-08, 1.7423577247654976e-08, 5.811278214196136e-08, 3.9313878374969136e-08, 2.3690166273127033e-08, 1.792834680713895e-08, 1.4406491999352511e-08, 7.0197305745978156e-09, 6.041436506620812e-08, 4.867428771149207e-08, 1.0685186957459791e-08, 1.005112082452797e-08, 4.2425732260653604e-08, 2.6077879411445792e-08, 5.136626679558276e-09, 1.6972753202892932e-09, 1.9131350725842822e-08, 2.1117190840413969e-07, 1.3372530638378066e-08, 2.000219723186092e-08, 1.0454146348592985e-07, 2.814382682458927e-08, 2.134452614654947e-07, 2.104584950757271e-08, 1.9114216343041335e-07, 3.957645233754192e-08, 2.9315843974586642e-08, 1.1051893977764848e-07, 4.840025980001883e-08, 5.583537188158516e-08, 1.2129960680340446e-07, 1.7761975001917563e-08, 2.561053769056798e-08, 1.2038481145282276e-07, 4.6748018789344314e-09, 1.591587306393194e-08, 3.147561500327356e-08, 6.147622144681009e-08, 2.204614979461894e-08, 3.2577264619249036e-07, 1.198899752807847e-07, 2.3818718375423487e-07, 1.490982981522393e-08, 5.1016343149967724e-08, 5.5141523347401744e-08, 2.288526701477167e-08, 5.7140397302646306e-08, 5.185041231925425e-07, 4.977233558633998e-08, 1.104976110610778e-08, 1.26408238543263e-07, 8.174780674607973e-08]]} [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_paddle.py::test_paddle_runtime [e2e-predictor] predictor/test_paddle.py::test_paddle_v2_kserve 2026-07-28 17:25:32.863 4494 kserve.trace Host from isvc status URL = isvc-paddle-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:25:32.863 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-paddle-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:25:32.863 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:25:32.863 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:25:32.863 4494 kserve.trace Using external route host: isvc-paddle-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:25:32.863 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-paddle-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:25:32.909 4494 kserve.trace Sending Header = {'Host': 'isvc-paddle-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:25:32.909 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-paddle-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:25:32.910 4494 kserve.trace base url = https://isvc-paddle-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:25:32.910 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-paddle-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:25:34.645 4494 kserve.trace Got response "id": "ca5446de-c181-4dc9-81f2-9b80dc3d5a00","model_name": "isvc-paddle-v2-kserve","outputs": ["name": "output-0","shape": [1, 1000],"datatype": "FP32","data": [6.736569968524009e-09, 1.153582651625129e-08, 5.142186410012073e-08, 6.647101002954514e-08, 4.094442118685038e-08, 1.3402272713847196e-07, 9.35544619551365e-08, 2.8935589924117266e-08, 6.845294819868286e-08, 7.680520752728626e-08, 2.0334418877610005e-06, 1.1085519417974865e-06, 2.3477346644540376e-07, 6.581955744877632e-07, 0.00012372950732242316, 4.2878312456196e-07, 6.419879809982376e-06, 0.9993487000465393, 7.371911488007754e-05, 3.101079710177146e-05, 5.602713827101979e-06, 2.1862256289750803e-06, 1.9543842100233633e-08, 3.728848128048412e-07, 4.29031416615544e-07, 1.8250955235998845e-07, 7.159837878134567e-08, 9.231503561579757e-09, 6.469168170042394e-07, 7.031603299623157e-09, 4.4511686780879245e-08, 1.2455829789814743e-07, 9.446210924579646e-08, 4.3476589439706004e-08, 4.6581672563661414e-07, 6.797636586952649e-08, 2.1060014887552825e-07, 2.2604888627597575e-08, 1.4311326879123953e-07, 7.951216218771151e-08, 1.2341666888460168e-07, 1.0921577313638409e-06, 1.5243762391037308e-05, 3.1172930903267115e-07, 2.415180802017858e-07, 6.863702850523623e-08, 8.46760883632669e-08, 9.424560687421035e-08, 1.0219084245477461e-08, 3.3770398033539095e-08, 3.6928447855189006e-08, 1.3693848188722768e-07, 1.067415098532365e-07, 2.5994486918534676e-07, 3.486601087843155e-07, 3.132020509610811e-08, 3.5748260529544496e-07, 6.648343742199359e-08, 3.163856945320731e-07, 1.2095728152416996e-06, 8.663983663836916e-08, 4.014367860349921e-08, 1.2544606420306081e-07, 3.320077013313494e-08, 1.973121896980956e-07, 3.8063546980993124e-07, 1.382770449254167e-07, 2.3001977211833946e-08, 7.14414980507172e-08, 2.851079372589993e-08, 2.982524804906461e-08, 8.935905526641363e-08, 6.223806394700659e-07, 6.478746428228987e-08, 1.3662854314588913e-07, 9.973038572752557e-08, 2.576123669939534e-08, 4.130169273253159e-08, 3.9383902361578293e-08, 1.2157941853274679e-07, 4.302776233089389e-06, 1.8178942582380841e-06, 1.8520014464229462e-06, 1.6245859342234326e-06, 1.64481571118813e-05, 1.0544756150920875e-05, 3.993004611402284e-06, 2.646454788646224e-07, 1.919327405630611e-05, 4.803187607649306e-07, 1.6962640358997305e-07, 4.550436642603017e-06, 4.2358526116004214e-05, 4.443309080670588e-06, 5.103951934870565e-06, 1.3506267350749113e-05, 4.1758286783988297e-07, 4.4944314936401497e-07, 3.156656873670727e-07, 1.0557428140600678e-06, 1.3364497952750298e-08, 1.3893487249561076e-08, 6.770295613023336e-08, 1.4129548731034447e-07, 7.170429228153807e-08, 7.934354329108828e-08, 2.6391164809069778e-08, 2.6133946562367782e-08, 7.1966232972897615e-09, 2.1752097012495142e-08, 6.684530973188885e-08, 3.4177467966856057e-08, 1.6228092647452286e-07, 4.107051836399478e-07, 6.472068889706861e-07, 2.9513313393181306e-07, 5.653383094994524e-09, 4.8300755395302986e-08, 8.887371727439586e-09, 3.7305639466467255e-08, 1.7784008932153483e-08, 4.641856232723285e-09, 3.413062898971475e-08, 1.9373679549516964e-07, 1.2980041219634586e-06, 3.5640496776068176e-08, 2.1493017143825455e-08, 3.055253330330743e-07, 1.553226809392072e-07, 1.4520828699460253e-06, 3.488421285169352e-08, 3.825387466349639e-05, 4.508809183789708e-07, 4.176648928932991e-07, 6.770537765987683e-07, 1.4142071336209483e-07, 1.423571211489616e-05, 6.293736873885791e-07, 4.762812295666663e-06, 9.024806217894366e-07, 9.058892942448438e-07, 1.5713549146312289e-06, 1.5720408441666223e-07, 1.8185124872616143e-07, 7.193072093514274e-08, 1.1952631666645175e-06, 8.874753234522359e-07, 2.0870552930318809e-07, 9.906116105184992e-08, 7.793509837483725e-09, 1.0058372623689138e-07, 4.205883783470199e-07, 1.8436001880672848e-07, 1.6437729755125474e-07, 7.02525184692604e-08, 2.5704193262754416e-07, 7.586770323086967e-08, 7.841231308702845e-07, 2.495280568837188e-07, 5.157617977147311e-08, 4.0673629797538524e-08, 7.531674839356128e-09, 4.797425745550754e-08, 1.7419722553313477e-08, 1.7958475950763386e-07, 1.2566235163546935e-08 [e2e-predictor] , 8.975329990335013e-08, 3.269607518063822e-08, 1.1208221195602164e-07, 3.906690437815996e-08, 4.676846998563633e-08, 1.852334321483795e-07, 1.4832883721282997e-07, 1.2279221550670627e-07, 1.0729002042353386e-06, 3.6537970427019673e-09, 1.6198204377815273e-07, 1.6190487173162182e-08, 1.2004819893718377e-07, 1.480009448329156e-08, 4.022906807676918e-08, 2.1505718450498534e-07, 1.192555032503151e-07, 4.898190297808469e-08, 7.608841201545147e-08, 2.3137237903370078e-08, 8.52094430570105e-08, 9.586112525994395e-08, 1.3351485961266008e-07, 3.0216618540634954e-08, 4.4238134933038964e-08, 2.610634552979718e-08, 2.397684113475407e-07, 1.3192399705985736e-07, 1.6734691854480843e-08, 1.5883161097463017e-07, 4.064300753725547e-07, 8.753363545110915e-08, 8.366871497855755e-07, 3.437562767771851e-08, 7.847781802183817e-08, 8.526320982582547e-09, 9.601245665180613e-08, 5.258879127723048e-07, 1.3557291822507977e-07, 1.0307118003538562e-07, 1.0429684671464656e-08, 5.187630236491714e-08, 2.1869702493404475e-08, 1.179131725592697e-08, 2.9806198398318884e-08, 4.3383401759911067e-08, 2.999067660880428e-08, 2.8507258775789523e-08, 3.058627484620047e-08, 6.441018740588333e-08, 1.5363882610586188e-08, 1.597368459727022e-08, 2.57365329048298e-08, 1.090363141997841e-07, 3.211827603877282e-08, 6.819658615597746e-09, 1.9251054084179486e-07, 5.825749838095362e-08, 1.876556723345857e-07, 4.0070369777822634e-07, 1.5791380647556252e-08, 1.9501307235714194e-07, 1.0141919304373914e-08, 2.744775962071344e-08, 1.2843395680306458e-08, 3.7296960186949946e-08, 7.407404467585366e-08, 4.206033921150265e-08, 1.6924593282396927e-08, 1.4591853414458456e-07, 4.344931525679385e-08, 1.7191209167322086e-07, 3.5816999144344663e-08, 8.440128596021168e-09, 4.1947775031303536e-08, 2.513996122388562e-08, 2.8339847801817086e-08, 8.747072399728495e-08, 8.277008411994302e-09, 1.1676148936601294e-08, 1.4548319882123906e-07, 7.200192886358536e-09, 2.623576165206032e-06, 5.675644842995098e-07, 1.9483265987219056e-06, 6.752511438889996e-08, 8.168359499904909e-08, 1.0932910754490877e-07, 1.6708914074570203e-07, 3.138676518688044e-08, 2.9734879447573803e-08, 5.7520821883372264e-08, 5.8507939826313304e-08, 3.2544284067625995e-07, 3.330167075432655e-08, 4.186742614820105e-07, 1.5085691984495497e-07, 2.334644193524582e-07, 2.8639897209359333e-07, 2.2940034227758588e-07, 1.8537357959758083e-07, 3.151756686747831e-07, 1.1075776455982123e-06, 1.5369606387594104e-07, 1.9237216974943294e-07, 1.6404264613356645e-07, 2.900799245253438e-07, 1.2466382770526252e-07, 5.802550617772795e-08, 5.1861558603150115e-08, 6.009345909774311e-09, 1.23330778478703e-07, 1.3798263864828186e-07, 1.7370017246776115e-07, 5.617696388071636e-07, 5.160393001801822e-08, 4.813227150179955e-08, 8.032599652096906e-08, 2.0644911273848265e-06, 5.638538027596951e-07, 8.794099244369136e-07, 3.4785582556651207e-06, 2.972355162000895e-07, 3.390414349269122e-07, 9.468957529179534e-08, 3.754799493549399e-08, 1.5678830322940485e-07, 8.2030041426151e-08, 6.847877820348458e-09, 1.825139683830912e-08, 6.050154155445853e-08, 3.9562941367421445e-08, 1.0699794472657231e-07, 3.256623699599004e-07, 3.536899839673424e-07, 7.326217854597417e-08, 4.857605517827324e-07, 7.717640073678922e-07, 3.456741737295488e-08, 3.2461647947457095e-07, 3.1608240078639938e-06, 5.330938535053065e-08, 3.645648973815696e-07, 5.481521156980307e-07, 4.623012372917401e-08, 1.346600981833035e-07, 4.352903104631878e-08, 1.6403919289587066e-07, 2.4636646500653114e-08, 5.958662185889807e-08, 9.493569308460792e-08, 5.5234050222452424e-08, 5.741187081298449e-08, 1.1850225746457e-05, 5.826300366607029e-06, 7.420818292303011e-06, 9.127862199420633e-07, 2.001935399675858e-06, 1.0334889566365746e-06, 3.514648483360361e-08, 2.0589775431290036e-06, 3.5655068586493144e-07, 6.873136015883574e-08, 2.193502490754895e-09, 5.560284321859399e-08, 3.3266616128457827e-07, 1.3073517379780242e-07, 2.7187292417352182e-08, 1.0462799515664756e-08, 7.466601914529747e-07, 6.923080064780152e-08, 1.6145493830777013e-08, 8.568415310605815e-09, 4.762142236103273e-09, 1.233958357715892e-07, [e2e-predictor] 8.340525603500737e-09, 3.2648699299642203e-09, 5.063196084620358e-09, 4.0704488668552585e-09, 1.2043389396865223e-08, 5.1055555339019065e-09, 7.267066948202228e-09, 1.1844983305309142e-07, 7.53547197973603e-08, 6.386873252495207e-08, 1.621267564644313e-08, 2.6103893446816073e-07, 6.979319664424111e-07, 6.647392325476176e-08, 7.717375183347031e-07, 6.651137596236367e-07, 3.3244478458982485e-07, 3.707229154770175e-07, 3.995585302618565e-07, 6.411539743567118e-08, 7.107263400030206e-08, 1.6379799205878953e-07, 6.876715730186334e-08, 3.4624252975845593e-07, 2.0256192101442139e-07, 6.19600768914097e-07, 2.6840686118134727e-08, 6.720200644849683e-07, 1.1348156476742588e-06, 1.8397651047052932e-06, 6.397185075002199e-07, 7.257429501805746e-08, 4.2213261508550204e-07, 3.965735402289283e-07, 1.4037237860975438e-07, 3.249813573802385e-07, 1.5857477819736232e-07, 1.1122037335553614e-07, 7.39131564841955e-08, 3.4231769063808315e-07, 5.39788445053091e-08, 8.517191218970765e-08, 4.060975243191933e-06, 1.4478493540082127e-05, 7.316928396505773e-09, 6.948383646232514e-09, 4.4688626132938225e-08, 9.23126250995665e-08, 5.41190452452156e-08, 2.224259958438779e-07, 1.7609368185844687e-08, 2.0905980946395175e-08, 3.6797160873902612e-09, 6.177843658861093e-08, 1.7920085326750268e-07, 2.6278852871541858e-08, 2.698776313536655e-08, 1.6432605320915172e-07, 1.2827440798446332e-07, 4.468845560268164e-08, 6.316474099321567e-08, 1.9461522171582146e-08, 2.0871000927513705e-08, 2.2414262446091016e-08, 2.4764982953229264e-08, 6.785314532464781e-09, 2.4248494767675766e-08, 4.554914667664889e-09, 2.8976625543464252e-08, 2.0402152856036082e-08, 1.6009290959573264e-07, 2.019951637066697e-07, 1.6111656009343278e-08, 5.964040639128143e-08, 4.0982603977113286e-09, 3.907972612182675e-08, 7.511245847524606e-09, 5.965555942566425e-07, 1.6478051634294388e-07, 1.410681527147517e-08, 3.285524030616216e-08, 3.338656018314623e-09, 1.2200259469352659e-08, 4.624581961820695e-08, 6.842250765970448e-09, 1.7426071252657493e-08, 4.661075436729334e-08, 9.33183343931887e-08, 1.230591379908219e-07, 1.2359244294657401e-08, 1.1173082903326303e-08, 2.772451956900568e-08, 2.4191731995415466e-07, 3.4511506896706123e-07, 2.593734649281032e-08, 9.964443847820803e-08, 9.797671118860762e-09, 1.90853270964908e-07, 3.9726497647052383e-08, 2.6638923955601967e-08, 6.8740502179309715e-09, 3.146955052102385e-08, 2.4086276084744895e-07, 1.3126738451774145e-07, 2.1254035686979478e-07, 2.0501781605730685e-08, 3.694923123020999e-08, 6.563101351275691e-07, 2.5600135344916453e-08, 2.688264899575188e-08, 6.880517844365386e-07, 2.0092501529234141e-07, 2.7880052044793047e-08, 2.62837644982028e-08, 5.167780159354152e-08, 1.893516099471526e-07, 4.6184757707123936e-07, 1.1086663143089481e-08, 1.4542384008109366e-07, 2.8736709012378014e-08, 6.105074703555147e-07, 1.2016337258557996e-08, 1.3944533350240818e-07, 2.0936788303060894e-08, 4.3800788773751265e-08, 1.9662968142597492e-08, 6.6543535126584175e-09, 1.1149432488366529e-08, 6.424846787922434e-08, 6.971495380980741e-09, 3.2599853927450795e-09, 1.4260012726197147e-08, 2.7894840215481054e-08, 8.115682703646598e-08, 2.599534454361674e-08, 2.285521105704902e-08, 1.05594978094814e-07, 8.145473628928812e-08, 3.779314994289962e-08, 4.8818307618603285e-08, 2.3420332340151617e-08, 1.0599223720930695e-08, 3.604067799756194e-08, 5.062377184117395e-08, 3.6804056691153164e-08, 1.5015616527946918e-09, 1.4474869658442913e-06, 1.0761959856608883e-06, 1.304976251503831e-07, 3.073563448197092e-08, 1.7183838352252678e-08, 2.042087920983704e-08, 7.904924714807748e-09, 1.6901843480354728e-07, 1.2416376371504612e-08, 5.4757610001843204e-08, 2.6250097207025647e-08, 1.3261202802539174e-08, 6.298007804161898e-08, 1.2709827856838274e-08, 2.0171410142211244e-07, 4.386575014336813e-08, 6.962276444255622e-08, 2.956475668725034e-07, 7.925040677037032e-07, 2.0868660044470744e-07, 1.7341581326490996e-07, 4.294189182019181e-08, 4.213362547744737e-09, 8.8246935092684e-08, 1.7341355018629656e-08, 7.321497008661026e-08, 1.7941481189609476e-08, 1.1245008835203407e-07, 4.2423405943736725 [e2e-predictor] e-07, 8.259470440918903e-09, 1.1336252470073305e-07, 8.268681739309613e-08, 2.2186704029536486e-08, 1.9539518447686532e-08, 1.0675572426066537e-08, 3.288476690954667e-08, 2.4340661042288048e-08, 6.639029948019015e-08, 5.604618547039308e-09, 1.386590220420203e-08, 6.6757905869963e-08, 1.1355724360839758e-08, 3.1321272331297223e-07, 3.1244848486267074e-08, 1.502163371469578e-07, 1.346161226933873e-08, 1.8882738572756352e-07, 4.645693962856967e-08, 4.645321638463429e-08, 7.71435715307689e-09, 3.585664032357272e-08, 7.608256247237932e-09, 4.2214487905312126e-08, 4.3406713778892936e-09, 1.3401408871516196e-08, 8.565495335233209e-08, 1.7045202582721686e-08, 5.422122839604526e-08, 3.0218750168842234e-08, 6.153300091682468e-08, 3.9388163841636015e-09, 4.135585029985123e-08, 1.7818981845607595e-08, 4.31053557292671e-08, 3.9032919119108556e-09, 7.663005163749403e-08, 1.1890258555524724e-08, 9.304103265606045e-09, 1.0967905472725192e-09, 1.0536636452229686e-08, 1.1516662112853737e-07, 8.134422273542441e-07, 5.9525618212319387e-08, 2.8063100288022724e-08, 1.283296402476708e-08, 1.060557863752365e-07, 7.872866945035639e-07, 2.7501052102252288e-08, 3.936232229051484e-09, 2.0519189902756807e-08, 7.394696410756296e-09, 3.5983390489491285e-08, 2.5378296797384792e-08, 4.6989228508209635e-08, 7.549436631393291e-09, 6.322734407149255e-07, 5.581916262542563e-09, 1.2963950268840563e-07, 1.587476106124086e-08, 3.3837391555380236e-08, 6.474394353972457e-09, 9.121036015358186e-08, 1.3918366015275296e-08, 8.229938508463874e-09, 2.7060956142577197e-08, 2.6095642979839795e-08, 5.722282203635132e-09, 6.963402938708896e-07, 4.685033871965061e-08, 9.590480409826796e-09, 2.0991818416860042e-07, 3.082116961650172e-08, 3.563118511351604e-08, 7.326215154535021e-07, 2.1759417450084584e-06, 2.4074907400972734e-07, 2.974484800688515e-07, 2.528977205429328e-08, 7.667869894589785e-09, 2.6632636718204594e-07, 3.435845030708151e-08, 2.3129958393042216e-08, 3.1239306252928145e-08, 2.8691238185274415e-07, 3.8957892911639647e-08, 2.4183783864373254e-08, 1.158230222841894e-08, 5.1544706280992614e-08, 2.0343204454320585e-08, 8.201909196259294e-08, 1.1641392738681589e-08, 5.496298740581551e-07, 1.1682007361457636e-08, 4.7576321549058775e-08, 1.6349654075042963e-08, 4.090812311119407e-08, 2.1270925287808495e-07, 1.669703095785735e-07, 3.989651631286506e-08, 2.8524043500510743e-06, 1.2500227342115977e-07, 2.4846258384059183e-07, 1.2454135500661323e-08, 2.9699904757762852e-08, 4.25093871214699e-09, 1.6144196024470148e-07, 2.6385819751340023e-07, 7.637975585339518e-09, 3.4455236441743864e-09, 7.273212787595185e-08, 1.763124757303558e-08, 7.586505645917896e-09, 2.154678924171094e-08, 1.267521554382256e-07, 2.5636829548147944e-08, 3.500932521660616e-08, 6.472151170555662e-08, 8.387794458997178e-09, 3.0694742747527926e-08, 7.520281286588215e-08, 1.5724816648798878e-07, 1.9633705505839316e-07, 1.2290654183289007e-07, 1.1121071841202479e-09, 1.5468767600168576e-08, 9.916888643601851e-09, 6.882375487293757e-07, 8.267530660077682e-08, 4.46946799570469e-08, 2.075171856574798e-08, 8.649272587035739e-08, 5.20271186132959e-08, 4.5564465978031876e-08, 2.031970502969216e-08, 8.705073994974555e-09, 6.45198596771479e-08, 2.1777212921847422e-08, 1.0309374687267336e-08, 3.2118705917127954e-08, 2.3336649235261575e-07, 8.053995692591798e-09, 1.9623111313649133e-07, 1.2887916511772346e-07, 1.5392334518082862e-08, 1.401880500573327e-09, 5.696758975659577e-08, 6.079961423210989e-09, 1.0782660986308201e-08, 2.4260382147645032e-08, 1.9388348704296732e-08, 2.2969983604070876e-07, 1.997151066746028e-08, 2.8477696645268225e-08, 5.227270349905666e-08, 2.739241153904004e-07, 9.85714976309282e-08, 3.129077086327925e-08, 4.1513992243835673e-08, 5.2511412995670526e-09, 1.5806674582563573e-06, 8.547505672140687e-07, 1.0688979479311911e-08, 1.0621677120070672e-06, 7.737217089243131e-08, 6.394137130882882e-08, 1.1698201518584028e-07, 1.0486479595783749e-07, 2.1160776952910965e-07, 1.5339432835048683e-08, 5.094390331805698e-08, 1.4005206772083056e-08, 2.6281709253339613e-08, 8.77830927947798e-08, 7.77198572 [e2e-predictor] 1660712e-09, 4.228822803042931e-08, 3.3243432540075446e-07, 7.72913466562386e-08, 7.63679175452836e-10, 5.989415541307608e-08, 1.3260753917165857e-07, 1.2853475084284582e-07, 8.844149412823299e-09, 1.0194228394766469e-07, 2.4937534703894926e-07, 1.6547782877296413e-07, 1.176258734858493e-08, 1.1496064900029523e-07, 2.934237670615403e-07, 1.3261050924029405e-08, 8.630173198298507e-08, 5.739413211358624e-08, 1.1093964502606468e-07, 2.2933429022486962e-07, 3.470581191322708e-07, 1.475092403779854e-07, 1.5024763655446804e-08, 6.454251177956394e-08, 5.164459437878577e-08, 6.23732674398525e-08, 1.2935890936205396e-07, 1.4051898311606692e-08, 5.386868906498421e-08, 2.082729722019394e-08, 1.3040525104202061e-08, 1.05788906523685e-07, 1.507953939494655e-08, 8.926216423787992e-07, 4.637372086335745e-08, 7.480928161385236e-07, 5.883074472023964e-09, 2.870727655590599e-09, 8.381493898923509e-07, 7.341868002441743e-09, 1.4245849833116608e-08, 1.0926260785026898e-07, 1.1308060265946551e-07, 2.52335809136639e-07, 1.178269073420779e-07, 4.667730024721095e-08, 2.79587997198405e-09, 3.436349871321909e-08, 1.4674341741738317e-07, 3.539640758276619e-08, 2.0580911552769976e-07, 7.18380377406902e-08, 2.7693602078215918e-08, 4.5492825506698864e-08, 1.955898198957584e-08, 1.5359576721607482e-08, 1.2336422550163206e-08, 2.957029643368969e-08, 2.877517033539334e-07, 9.051751135302766e-07, 2.3732353326977318e-07, 1.6521470413977113e-08, 1.5478654802336678e-08, 3.526755776306345e-08, 3.61636587342673e-08, 1.615889466677345e-08, 7.649993705172164e-08, 1.9661243300106435e-08, 4.9174737171142624e-08, 1.1712456426948847e-07, 1.0889120716228717e-08, 1.4941074368834961e-06, 1.0185750909386115e-08, 3.7575578204496196e-08, 2.0970718495050278e-08, 3.368522527580353e-08, 4.845519985252622e-09, 6.039555842107802e-07, 1.0373183201295433e-08, 2.841620698745828e-07, 4.499007388858445e-07, 3.4631430168019506e-08, 7.720588968140873e-08, 1.4711066853578814e-07, 1.160143270340086e-07, 4.0074306184578745e-07, 3.0256124716743216e-08, 6.706726196625823e-08, 2.012845534693497e-08, 1.5987545731732666e-09, 4.1919221871467016e-08, 1.3166965295852151e-08, 3.231780709711529e-08, 9.247563781400459e-08, 1.3075164417841734e-07, 1.0574149911235509e-07, 3.762125899697821e-08, 1.0942090256094161e-07, 7.001387558602801e-08, 2.7426720450307585e-08, 2.0766329100752046e-08, 4.5402845927355884e-08, 3.390360703292572e-08, 1.0469531730450399e-07, 2.8271228913467894e-08, 3.406190955956845e-07, 5.146158059687878e-07, 6.740644380442973e-07, 6.3821934404018066e-09, 3.638748680145909e-08, 3.6260146885069844e-08, 1.6065344254911906e-07, 3.639007957190188e-07, 6.232625970881145e-09, 4.8054303647404595e-08, 3.3725854109434295e-08, 6.3287848206528e-07, 6.480550496235082e-08, 2.116489525860743e-07, 8.38768769995113e-08, 1.758889212055692e-08, 2.729004577872729e-09, 2.144768451728396e-08, 7.861175532752895e-08, 2.011793931444572e-08, 2.840733515085958e-08, 2.492219834948628e-07, 2.0156347702027233e-08, 2.6551370879701608e-08, 2.7847898209643063e-08, 6.907038496706264e-09, 1.8805204504701578e-08, 1.3006692611838844e-08, 3.685865976876812e-07, 3.9678877783444477e-07, 2.7591790185965692e-08, 2.5228539257682314e-08, 1.5469838388071366e-07, 3.6892611632310945e-08, 1.4401622117077295e-09, 2.1504664715621402e-08, 5.068058328561165e-08, 5.081657761252245e-08, 1.1718628734058711e-08, 5.409200909411993e-08, 7.138175419640902e-07, 2.523685509459028e-07, 7.071956531490287e-08, 7.199660245760242e-08, 1.2525339876390262e-08, 3.4802988579940575e-07, 1.9591639954796847e-07, 1.2404500182583433e-07, 1.2346009725661133e-07, 1.92011366806355e-08, 1.9895020386684337e-07, 3.7876212388709973e-08, 1.0629652713589621e-08, 1.2436973229057458e-08, 2.186162078032794e-07, 2.618108396745811e-07, 1.1128912547064829e-07, 1.0776497560982534e-07, 6.380246109216614e-09, 3.895029632161595e-09, 1.576254788915321e-07, 2.9089968212048234e-09, 1.03809227880447e-08, 2.8134863327977655e-08, 1.0777869263733919e-08, 1.3605805371241786e-08, 2.9236161935841665e-08, 1.918957508451058e-07, 2.1994748067299952e-07, 1.3263833587018325e-08, 4.900423 [e2e-predictor] 533626963e-08, 2.9808058243929736e-09, 8.925934658066126e-09, 1.0996840771326788e-08, 7.715520666806697e-09, 7.454385553273823e-09, 5.0860990086221136e-08, 1.5129604946650943e-07, 1.1959995660504319e-08, 1.1323217030678734e-08, 9.39119804144184e-09, 9.585582461113518e-08, 1.9055129030220996e-08, 1.8105044574667772e-08, 6.179151057494892e-08, 6.389322493305372e-08, 1.1853056314237165e-08, 9.372677745034252e-09, 1.2332006349424773e-07, 1.6521848777983905e-08, 1.246099685658919e-07, 4.196111902388111e-09, 3.9965367903960214e-08, 1.255440107428285e-08, 1.4302015571843185e-08, 6.631698745707126e-09, 5.964162053118116e-09, 5.556878956980427e-09, 1.419230528654225e-07, 1.7612896030527736e-08, 3.3801444487835397e-07, 7.856438344333583e-08, 2.9667463152804885e-08, 2.8991748877160717e-06, 1.3787196166958893e-06, 5.313577133847502e-09, 2.5128217728820346e-08, 8.406522056247923e-08, 4.492775573794461e-08, 5.4087468726038423e-08, 2.4238921980668238e-08, 4.0167356019082945e-07, 4.1083040969169815e-08, 5.415281023601892e-08, 8.640659743264223e-09, 5.773174294176897e-08, 2.6442751277500065e-07, 8.953106203080097e-07, 2.799365539374321e-08, 5.889395637836969e-09, 4.178830081968954e-08, 2.888028660663622e-08, 2.1357223545237503e-08, 2.3023956430279213e-07, 8.786517469161481e-08, 2.069703963769598e-09, 2.2363829899063603e-08, 3.2032425600903025e-09, 1.176860120466472e-08, 6.963498577761129e-08, 2.2719042647167953e-08, 7.360319465021803e-09, 6.922469264480924e-09, 3.2138252947788715e-08, 1.3705631829452614e-07, 1.9814805796158907e-08, 1.0578844467090676e-08, 2.7049035011827982e-08, 2.9755513608620277e-09, 2.1773475467057324e-08, 1.0975404585167325e-08, 1.9918440230526357e-08, 2.3881801425318372e-08, 2.1357100976615584e-08, 6.109046157831699e-09, 1.1890372242362446e-08, 1.1459750481890296e-08, 3.731681275098708e-08, 1.5726065782928345e-08, 3.403987491878979e-08, 3.6921193213856895e-08, 9.281668411631472e-08, 1.2323060616381554e-07, 4.234707162709128e-08, 1.7423577247654976e-08, 5.811278214196136e-08, 3.9313878374969136e-08, 2.3690166273127033e-08, 1.792834680713895e-08, 1.4406491999352511e-08, 7.0197305745978156e-09, 6.041436506620812e-08, 4.867428771149207e-08, 1.0685186957459791e-08, 1.005112082452797e-08, 4.2425732260653604e-08, 2.6077879411445792e-08, 5.136626679558276e-09, 1.6972753202892932e-09, 1.9131350725842822e-08, 2.1117190840413969e-07, 1.3372530638378066e-08, 2.000219723186092e-08, 1.0454146348592985e-07, 2.814382682458927e-08, 2.134452614654947e-07, 2.104584950757271e-08, 1.9114216343041335e-07, 3.957645233754192e-08, 2.9315843974586642e-08, 1.1051893977764848e-07, 4.840025980001883e-08, 5.583537188158516e-08, 1.2129960680340446e-07, 1.7761975001917563e-08, 2.561053769056798e-08, 1.2038481145282276e-07, 4.6748018789344314e-09, 1.591587306393194e-08, 3.147561500327356e-08, 6.147622144681009e-08, 2.204614979461894e-08, 3.2577264619249036e-07, 1.198899752807847e-07, 2.3818718375423487e-07, 1.490982981522393e-08, 5.1016343149967724e-08, 5.5141523347401744e-08, 2.288526701477167e-08, 5.7140397302646306e-08, 5.185041231925425e-07, 4.977233558633998e-08, 1.104976110610778e-08, 1.26408238543263e-07, 8.174780674607973e-08],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] 2026-07-28 17:25:34.645 4494 kserve.trace INFO [utils.py:predict():148] Got response "id": "ca5446de-c181-4dc9-81f2-9b80dc3d5a00","model_name": "isvc-paddle-v2-kserve","outputs": ["name": "output-0","shape": [1, 1000],"datatype": "FP32","data": [6.736569968524009e-09, 1.153582651625129e-08, 5.142186410012073e-08, 6.647101002954514e-08, 4.094442118685038e-08, 1.3402272713847196e-07, 9.35544619551365e-08, 2.8935589924117266e-08, 6.845294819868286e-08, 7.680520752728626e-08, 2.0334418877610005e-06, 1.1085519417974865e-06, 2.3477346644540376e-07, 6.581955744877632e-07, 0.00012372950732242316, 4.2878312456196e-07, 6.419879809982376e-06, 0.9993487000465393, 7.371911488007754e-05, 3.101079710177146e-05, 5.602713827101979e-06, 2.1862256289750803e-06, 1.9543842100233633e-08, 3.728848128048412e-07, 4.29031416615544e-07, 1.8250955235998845e-07, 7.159837878134567e-08, 9.231503561579757e-09, 6.469168170042394e-07, 7.031603299623157e-09, 4.4511686780879245e-08, 1.2455829789814743e-07, 9.446210924579646e-08, 4.3476589439706004e-08, 4.6581672563661414e-07, 6.797636586952649e-08, 2.1060014887552825e-07, 2.2604888627597575e-08, 1.4311326879123953e-07, 7.951216218771151e-08, 1.2341666888460168e-07, 1.0921577313638409e-06, 1.5243762391037308e-05, 3.1172930903267115e-07, 2.415180802017858e-07, 6.863702850523623e-08, 8.46760883632669e-08, 9.424560687421035e-08, 1.0219084245477461e-08, 3.3770398033539095e-08, 3.6928447855189006e-08, 1.3693848188722768e-07, 1.067415098532365e-07, 2.5994486918534676e-07, 3.486601087843155e-07, 3.132020509610811e-08, 3.5748260529544496e-07, 6.648343742199359e-08, 3.163856945320731e-07, 1.2095728152416996e-06, 8.663983663836916e-08, 4.014367860349921e-08, 1.2544606420306081e-07, 3.320077013313494e-08, 1.973121896980956e-07, 3.8063546980993124e-07, 1.382770449254167e-07, 2.3001977211833946e-08, 7.14414980507172e-08, 2.851079372589993e-08, 2.982524804906461e-08, 8.935905526641363e-08, 6.223806394700659e-07, 6.478746428228987e-08, 1.3662854314588913e-07, 9.973038572752557e-08, 2.576123669939534e-08, 4.130169273253159e-08, 3.9383902361578293e-08, 1.2157941853274679e-07, 4.302776233089389e-06, 1.8178942582380841e-06, 1.8520014464229462e-06, 1.6245859342234326e-06, 1.64481571118813e-05, 1.0544756150920875e-05, 3.993004611402284e-06, 2.646454788646224e-07, 1.919327405630611e-05, 4.803187607649306e-07, 1.6962640358997305e-07, 4.550436642603017e-06, 4.2358526116004214e-05, 4.443309080670588e-06, 5.103951934870565e-06, 1.3506267350749113e-05, 4.1758286783988297e-07, 4.4944314936401497e-07, 3.156656873670727e-07, 1.0557428140600678e-06, 1.3364497952750298e-08, 1.3893487249561076e-08, 6.770295613023336e-08, 1.4129548731034447e-07, 7.170429228153807e-08, 7.934354329108828e-08, 2.6391164809069778e-08, 2.6133946562367782e-08, 7.1966232972897615e-09, 2.1752097012495142e-08, 6.684530973188885e-08, 3.4177467966856057e-08, 1.6228092647452286e-07, 4.107051836399478e-07, 6.472068889706861e-07, 2.9513313393181306e-07, 5.653383094994524e-09, 4.8300755395302986e-08, 8.887371727439586e-09, 3.7305639466467255e-08, 1.7784008932153483e-08, 4.641856232723285e-09, 3.413062898971475e-08, 1.9373679549516964e-07, 1.2980041219634586e-06, 3.5640496776068176e-08, 2.1493017143825455e-08, 3.055253330330743e-07, 1.553226809392072e-07, 1.4520828699460253e-06, 3.488421285169352e-08, 3.825387466349639e-05, 4.508809183789708e-07, 4.176648928932991e-07, 6.770537765987683e-07, 1.4142071336209483e-07, 1.423571211489616e-05, 6.293736873885791e-07, 4.762812295666663e-06, 9.024806217894366e-07, 9.058892942448438e-07, 1.5713549146312289e-06, 1.5720408441666223e-07, 1.8185124872616143e-07, 7.193072093514274e-08, 1.1952631666645175e-06, 8.874753234522359e-07, 2.0870552930318809e-07, 9.906116105184992e-08, 7.793509837483725e-09, 1.0058372623689138e-07, 4.205883783470199e-07, 1.8436001880672848e-07, 1.6437729755125474e-07, 7.02525184692604e-08, 2.5704193262754416e-07, 7.586770323086967e-08, 7.841231308702845e-07, 2.495280568837188e-07, 5.157617977147311e-08, 4.0673629797538524e-08, 7.531674839356128e-09, 4.797425745550754e-08, 1.7419722553313477e-08, 1.79584759507633 [e2e-predictor] 86e-07, 1.2566235163546935e-08, 8.975329990335013e-08, 3.269607518063822e-08, 1.1208221195602164e-07, 3.906690437815996e-08, 4.676846998563633e-08, 1.852334321483795e-07, 1.4832883721282997e-07, 1.2279221550670627e-07, 1.0729002042353386e-06, 3.6537970427019673e-09, 1.6198204377815273e-07, 1.6190487173162182e-08, 1.2004819893718377e-07, 1.480009448329156e-08, 4.022906807676918e-08, 2.1505718450498534e-07, 1.192555032503151e-07, 4.898190297808469e-08, 7.608841201545147e-08, 2.3137237903370078e-08, 8.52094430570105e-08, 9.586112525994395e-08, 1.3351485961266008e-07, 3.0216618540634954e-08, 4.4238134933038964e-08, 2.610634552979718e-08, 2.397684113475407e-07, 1.3192399705985736e-07, 1.6734691854480843e-08, 1.5883161097463017e-07, 4.064300753725547e-07, 8.753363545110915e-08, 8.366871497855755e-07, 3.437562767771851e-08, 7.847781802183817e-08, 8.526320982582547e-09, 9.601245665180613e-08, 5.258879127723048e-07, 1.3557291822507977e-07, 1.0307118003538562e-07, 1.0429684671464656e-08, 5.187630236491714e-08, 2.1869702493404475e-08, 1.179131725592697e-08, 2.9806198398318884e-08, 4.3383401759911067e-08, 2.999067660880428e-08, 2.8507258775789523e-08, 3.058627484620047e-08, 6.441018740588333e-08, 1.5363882610586188e-08, 1.597368459727022e-08, 2.57365329048298e-08, 1.090363141997841e-07, 3.211827603877282e-08, 6.819658615597746e-09, 1.9251054084179486e-07, 5.825749838095362e-08, 1.876556723345857e-07, 4.0070369777822634e-07, 1.5791380647556252e-08, 1.9501307235714194e-07, 1.0141919304373914e-08, 2.744775962071344e-08, 1.2843395680306458e-08, 3.7296960186949946e-08, 7.407404467585366e-08, 4.206033921150265e-08, 1.6924593282396927e-08, 1.4591853414458456e-07, 4.344931525679385e-08, 1.7191209167322086e-07, 3.5816999144344663e-08, 8.440128596021168e-09, 4.1947775031303536e-08, 2.513996122388562e-08, 2.8339847801817086e-08, 8.747072399728495e-08, 8.277008411994302e-09, 1.1676148936601294e-08, 1.4548319882123906e-07, 7.200192886358536e-09, 2.623576165206032e-06, 5.675644842995098e-07, 1.9483265987219056e-06, 6.752511438889996e-08, 8.168359499904909e-08, 1.0932910754490877e-07, 1.6708914074570203e-07, 3.138676518688044e-08, 2.9734879447573803e-08, 5.7520821883372264e-08, 5.8507939826313304e-08, 3.2544284067625995e-07, 3.330167075432655e-08, 4.186742614820105e-07, 1.5085691984495497e-07, 2.334644193524582e-07, 2.8639897209359333e-07, 2.2940034227758588e-07, 1.8537357959758083e-07, 3.151756686747831e-07, 1.1075776455982123e-06, 1.5369606387594104e-07, 1.9237216974943294e-07, 1.6404264613356645e-07, 2.900799245253438e-07, 1.2466382770526252e-07, 5.802550617772795e-08, 5.1861558603150115e-08, 6.009345909774311e-09, 1.23330778478703e-07, 1.3798263864828186e-07, 1.7370017246776115e-07, 5.617696388071636e-07, 5.160393001801822e-08, 4.813227150179955e-08, 8.032599652096906e-08, 2.0644911273848265e-06, 5.638538027596951e-07, 8.794099244369136e-07, 3.4785582556651207e-06, 2.972355162000895e-07, 3.390414349269122e-07, 9.468957529179534e-08, 3.754799493549399e-08, 1.5678830322940485e-07, 8.2030041426151e-08, 6.847877820348458e-09, 1.825139683830912e-08, 6.050154155445853e-08, 3.9562941367421445e-08, 1.0699794472657231e-07, 3.256623699599004e-07, 3.536899839673424e-07, 7.326217854597417e-08, 4.857605517827324e-07, 7.717640073678922e-07, 3.456741737295488e-08, 3.2461647947457095e-07, 3.1608240078639938e-06, 5.330938535053065e-08, 3.645648973815696e-07, 5.481521156980307e-07, 4.623012372917401e-08, 1.346600981833035e-07, 4.352903104631878e-08, 1.6403919289587066e-07, 2.4636646500653114e-08, 5.958662185889807e-08, 9.493569308460792e-08, 5.5234050222452424e-08, 5.741187081298449e-08, 1.1850225746457e-05, 5.826300366607029e-06, 7.420818292303011e-06, 9.127862199420633e-07, 2.001935399675858e-06, 1.0334889566365746e-06, 3.514648483360361e-08, 2.0589775431290036e-06, 3.5655068586493144e-07, 6.873136015883574e-08, 2.193502490754895e-09, 5.560284321859399e-08, 3.3266616128457827e-07, 1.3073517379780242e-07, 2.7187292417352182e-08, 1.0462799515664756e-08, 7.466601914529747e-07, 6.923080064780152e-08, 1.6145493830777013e-08, 8.568415310605815e-09, 4.76214223610327 [e2e-predictor] 3e-09, 1.233958357715892e-07, 8.340525603500737e-09, 3.2648699299642203e-09, 5.063196084620358e-09, 4.0704488668552585e-09, 1.2043389396865223e-08, 5.1055555339019065e-09, 7.267066948202228e-09, 1.1844983305309142e-07, 7.53547197973603e-08, 6.386873252495207e-08, 1.621267564644313e-08, 2.6103893446816073e-07, 6.979319664424111e-07, 6.647392325476176e-08, 7.717375183347031e-07, 6.651137596236367e-07, 3.3244478458982485e-07, 3.707229154770175e-07, 3.995585302618565e-07, 6.411539743567118e-08, 7.107263400030206e-08, 1.6379799205878953e-07, 6.876715730186334e-08, 3.4624252975845593e-07, 2.0256192101442139e-07, 6.19600768914097e-07, 2.6840686118134727e-08, 6.720200644849683e-07, 1.1348156476742588e-06, 1.8397651047052932e-06, 6.397185075002199e-07, 7.257429501805746e-08, 4.2213261508550204e-07, 3.965735402289283e-07, 1.4037237860975438e-07, 3.249813573802385e-07, 1.5857477819736232e-07, 1.1122037335553614e-07, 7.39131564841955e-08, 3.4231769063808315e-07, 5.39788445053091e-08, 8.517191218970765e-08, 4.060975243191933e-06, 1.4478493540082127e-05, 7.316928396505773e-09, 6.948383646232514e-09, 4.4688626132938225e-08, 9.23126250995665e-08, 5.41190452452156e-08, 2.224259958438779e-07, 1.7609368185844687e-08, 2.0905980946395175e-08, 3.6797160873902612e-09, 6.177843658861093e-08, 1.7920085326750268e-07, 2.6278852871541858e-08, 2.698776313536655e-08, 1.6432605320915172e-07, 1.2827440798446332e-07, 4.468845560268164e-08, 6.316474099321567e-08, 1.9461522171582146e-08, 2.0871000927513705e-08, 2.2414262446091016e-08, 2.4764982953229264e-08, 6.785314532464781e-09, 2.4248494767675766e-08, 4.554914667664889e-09, 2.8976625543464252e-08, 2.0402152856036082e-08, 1.6009290959573264e-07, 2.019951637066697e-07, 1.6111656009343278e-08, 5.964040639128143e-08, 4.0982603977113286e-09, 3.907972612182675e-08, 7.511245847524606e-09, 5.965555942566425e-07, 1.6478051634294388e-07, 1.410681527147517e-08, 3.285524030616216e-08, 3.338656018314623e-09, 1.2200259469352659e-08, 4.624581961820695e-08, 6.842250765970448e-09, 1.7426071252657493e-08, 4.661075436729334e-08, 9.33183343931887e-08, 1.230591379908219e-07, 1.2359244294657401e-08, 1.1173082903326303e-08, 2.772451956900568e-08, 2.4191731995415466e-07, 3.4511506896706123e-07, 2.593734649281032e-08, 9.964443847820803e-08, 9.797671118860762e-09, 1.90853270964908e-07, 3.9726497647052383e-08, 2.6638923955601967e-08, 6.8740502179309715e-09, 3.146955052102385e-08, 2.4086276084744895e-07, 1.3126738451774145e-07, 2.1254035686979478e-07, 2.0501781605730685e-08, 3.694923123020999e-08, 6.563101351275691e-07, 2.5600135344916453e-08, 2.688264899575188e-08, 6.880517844365386e-07, 2.0092501529234141e-07, 2.7880052044793047e-08, 2.62837644982028e-08, 5.167780159354152e-08, 1.893516099471526e-07, 4.6184757707123936e-07, 1.1086663143089481e-08, 1.4542384008109366e-07, 2.8736709012378014e-08, 6.105074703555147e-07, 1.2016337258557996e-08, 1.3944533350240818e-07, 2.0936788303060894e-08, 4.3800788773751265e-08, 1.9662968142597492e-08, 6.6543535126584175e-09, 1.1149432488366529e-08, 6.424846787922434e-08, 6.971495380980741e-09, 3.2599853927450795e-09, 1.4260012726197147e-08, 2.7894840215481054e-08, 8.115682703646598e-08, 2.599534454361674e-08, 2.285521105704902e-08, 1.05594978094814e-07, 8.145473628928812e-08, 3.779314994289962e-08, 4.8818307618603285e-08, 2.3420332340151617e-08, 1.0599223720930695e-08, 3.604067799756194e-08, 5.062377184117395e-08, 3.6804056691153164e-08, 1.5015616527946918e-09, 1.4474869658442913e-06, 1.0761959856608883e-06, 1.304976251503831e-07, 3.073563448197092e-08, 1.7183838352252678e-08, 2.042087920983704e-08, 7.904924714807748e-09, 1.6901843480354728e-07, 1.2416376371504612e-08, 5.4757610001843204e-08, 2.6250097207025647e-08, 1.3261202802539174e-08, 6.298007804161898e-08, 1.2709827856838274e-08, 2.0171410142211244e-07, 4.386575014336813e-08, 6.962276444255622e-08, 2.956475668725034e-07, 7.925040677037032e-07, 2.0868660044470744e-07, 1.7341581326490996e-07, 4.294189182019181e-08, 4.213362547744737e-09, 8.8246935092684e-08, 1.7341355018629656e-08, 7.321497008661026e-08, 1.7941481189609476e-08, 1.1245008835 [e2e-predictor] 203407e-07, 4.2423405943736725e-07, 8.259470440918903e-09, 1.1336252470073305e-07, 8.268681739309613e-08, 2.2186704029536486e-08, 1.9539518447686532e-08, 1.0675572426066537e-08, 3.288476690954667e-08, 2.4340661042288048e-08, 6.639029948019015e-08, 5.604618547039308e-09, 1.386590220420203e-08, 6.6757905869963e-08, 1.1355724360839758e-08, 3.1321272331297223e-07, 3.1244848486267074e-08, 1.502163371469578e-07, 1.346161226933873e-08, 1.8882738572756352e-07, 4.645693962856967e-08, 4.645321638463429e-08, 7.71435715307689e-09, 3.585664032357272e-08, 7.608256247237932e-09, 4.2214487905312126e-08, 4.3406713778892936e-09, 1.3401408871516196e-08, 8.565495335233209e-08, 1.7045202582721686e-08, 5.422122839604526e-08, 3.0218750168842234e-08, 6.153300091682468e-08, 3.9388163841636015e-09, 4.135585029985123e-08, 1.7818981845607595e-08, 4.31053557292671e-08, 3.9032919119108556e-09, 7.663005163749403e-08, 1.1890258555524724e-08, 9.304103265606045e-09, 1.0967905472725192e-09, 1.0536636452229686e-08, 1.1516662112853737e-07, 8.134422273542441e-07, 5.9525618212319387e-08, 2.8063100288022724e-08, 1.283296402476708e-08, 1.060557863752365e-07, 7.872866945035639e-07, 2.7501052102252288e-08, 3.936232229051484e-09, 2.0519189902756807e-08, 7.394696410756296e-09, 3.5983390489491285e-08, 2.5378296797384792e-08, 4.6989228508209635e-08, 7.549436631393291e-09, 6.322734407149255e-07, 5.581916262542563e-09, 1.2963950268840563e-07, 1.587476106124086e-08, 3.3837391555380236e-08, 6.474394353972457e-09, 9.121036015358186e-08, 1.3918366015275296e-08, 8.229938508463874e-09, 2.7060956142577197e-08, 2.6095642979839795e-08, 5.722282203635132e-09, 6.963402938708896e-07, 4.685033871965061e-08, 9.590480409826796e-09, 2.0991818416860042e-07, 3.082116961650172e-08, 3.563118511351604e-08, 7.326215154535021e-07, 2.1759417450084584e-06, 2.4074907400972734e-07, 2.974484800688515e-07, 2.528977205429328e-08, 7.667869894589785e-09, 2.6632636718204594e-07, 3.435845030708151e-08, 2.3129958393042216e-08, 3.1239306252928145e-08, 2.8691238185274415e-07, 3.8957892911639647e-08, 2.4183783864373254e-08, 1.158230222841894e-08, 5.1544706280992614e-08, 2.0343204454320585e-08, 8.201909196259294e-08, 1.1641392738681589e-08, 5.496298740581551e-07, 1.1682007361457636e-08, 4.7576321549058775e-08, 1.6349654075042963e-08, 4.090812311119407e-08, 2.1270925287808495e-07, 1.669703095785735e-07, 3.989651631286506e-08, 2.8524043500510743e-06, 1.2500227342115977e-07, 2.4846258384059183e-07, 1.2454135500661323e-08, 2.9699904757762852e-08, 4.25093871214699e-09, 1.6144196024470148e-07, 2.6385819751340023e-07, 7.637975585339518e-09, 3.4455236441743864e-09, 7.273212787595185e-08, 1.763124757303558e-08, 7.586505645917896e-09, 2.154678924171094e-08, 1.267521554382256e-07, 2.5636829548147944e-08, 3.500932521660616e-08, 6.472151170555662e-08, 8.387794458997178e-09, 3.0694742747527926e-08, 7.520281286588215e-08, 1.5724816648798878e-07, 1.9633705505839316e-07, 1.2290654183289007e-07, 1.1121071841202479e-09, 1.5468767600168576e-08, 9.916888643601851e-09, 6.882375487293757e-07, 8.267530660077682e-08, 4.46946799570469e-08, 2.075171856574798e-08, 8.649272587035739e-08, 5.20271186132959e-08, 4.5564465978031876e-08, 2.031970502969216e-08, 8.705073994974555e-09, 6.45198596771479e-08, 2.1777212921847422e-08, 1.0309374687267336e-08, 3.2118705917127954e-08, 2.3336649235261575e-07, 8.053995692591798e-09, 1.9623111313649133e-07, 1.2887916511772346e-07, 1.5392334518082862e-08, 1.401880500573327e-09, 5.696758975659577e-08, 6.079961423210989e-09, 1.0782660986308201e-08, 2.4260382147645032e-08, 1.9388348704296732e-08, 2.2969983604070876e-07, 1.997151066746028e-08, 2.8477696645268225e-08, 5.227270349905666e-08, 2.739241153904004e-07, 9.85714976309282e-08, 3.129077086327925e-08, 4.1513992243835673e-08, 5.2511412995670526e-09, 1.5806674582563573e-06, 8.547505672140687e-07, 1.0688979479311911e-08, 1.0621677120070672e-06, 7.737217089243131e-08, 6.394137130882882e-08, 1.1698201518584028e-07, 1.0486479595783749e-07, 2.1160776952910965e-07, 1.5339432835048683e-08, 5.094390331805698e-08, 1.4005206772083056e-08, 2.6281709253339613e-08, 8. [e2e-predictor] 77830927947798e-08, 7.771985721660712e-09, 4.228822803042931e-08, 3.3243432540075446e-07, 7.72913466562386e-08, 7.63679175452836e-10, 5.989415541307608e-08, 1.3260753917165857e-07, 1.2853475084284582e-07, 8.844149412823299e-09, 1.0194228394766469e-07, 2.4937534703894926e-07, 1.6547782877296413e-07, 1.176258734858493e-08, 1.1496064900029523e-07, 2.934237670615403e-07, 1.3261050924029405e-08, 8.630173198298507e-08, 5.739413211358624e-08, 1.1093964502606468e-07, 2.2933429022486962e-07, 3.470581191322708e-07, 1.475092403779854e-07, 1.5024763655446804e-08, 6.454251177956394e-08, 5.164459437878577e-08, 6.23732674398525e-08, 1.2935890936205396e-07, 1.4051898311606692e-08, 5.386868906498421e-08, 2.082729722019394e-08, 1.3040525104202061e-08, 1.05788906523685e-07, 1.507953939494655e-08, 8.926216423787992e-07, 4.637372086335745e-08, 7.480928161385236e-07, 5.883074472023964e-09, 2.870727655590599e-09, 8.381493898923509e-07, 7.341868002441743e-09, 1.4245849833116608e-08, 1.0926260785026898e-07, 1.1308060265946551e-07, 2.52335809136639e-07, 1.178269073420779e-07, 4.667730024721095e-08, 2.79587997198405e-09, 3.436349871321909e-08, 1.4674341741738317e-07, 3.539640758276619e-08, 2.0580911552769976e-07, 7.18380377406902e-08, 2.7693602078215918e-08, 4.5492825506698864e-08, 1.955898198957584e-08, 1.5359576721607482e-08, 1.2336422550163206e-08, 2.957029643368969e-08, 2.877517033539334e-07, 9.051751135302766e-07, 2.3732353326977318e-07, 1.6521470413977113e-08, 1.5478654802336678e-08, 3.526755776306345e-08, 3.61636587342673e-08, 1.615889466677345e-08, 7.649993705172164e-08, 1.9661243300106435e-08, 4.9174737171142624e-08, 1.1712456426948847e-07, 1.0889120716228717e-08, 1.4941074368834961e-06, 1.0185750909386115e-08, 3.7575578204496196e-08, 2.0970718495050278e-08, 3.368522527580353e-08, 4.845519985252622e-09, 6.039555842107802e-07, 1.0373183201295433e-08, 2.841620698745828e-07, 4.499007388858445e-07, 3.4631430168019506e-08, 7.720588968140873e-08, 1.4711066853578814e-07, 1.160143270340086e-07, 4.0074306184578745e-07, 3.0256124716743216e-08, 6.706726196625823e-08, 2.012845534693497e-08, 1.5987545731732666e-09, 4.1919221871467016e-08, 1.3166965295852151e-08, 3.231780709711529e-08, 9.247563781400459e-08, 1.3075164417841734e-07, 1.0574149911235509e-07, 3.762125899697821e-08, 1.0942090256094161e-07, 7.001387558602801e-08, 2.7426720450307585e-08, 2.0766329100752046e-08, 4.5402845927355884e-08, 3.390360703292572e-08, 1.0469531730450399e-07, 2.8271228913467894e-08, 3.406190955956845e-07, 5.146158059687878e-07, 6.740644380442973e-07, 6.3821934404018066e-09, 3.638748680145909e-08, 3.6260146885069844e-08, 1.6065344254911906e-07, 3.639007957190188e-07, 6.232625970881145e-09, 4.8054303647404595e-08, 3.3725854109434295e-08, 6.3287848206528e-07, 6.480550496235082e-08, 2.116489525860743e-07, 8.38768769995113e-08, 1.758889212055692e-08, 2.729004577872729e-09, 2.144768451728396e-08, 7.861175532752895e-08, 2.011793931444572e-08, 2.840733515085958e-08, 2.492219834948628e-07, 2.0156347702027233e-08, 2.6551370879701608e-08, 2.7847898209643063e-08, 6.907038496706264e-09, 1.8805204504701578e-08, 1.3006692611838844e-08, 3.685865976876812e-07, 3.9678877783444477e-07, 2.7591790185965692e-08, 2.5228539257682314e-08, 1.5469838388071366e-07, 3.6892611632310945e-08, 1.4401622117077295e-09, 2.1504664715621402e-08, 5.068058328561165e-08, 5.081657761252245e-08, 1.1718628734058711e-08, 5.409200909411993e-08, 7.138175419640902e-07, 2.523685509459028e-07, 7.071956531490287e-08, 7.199660245760242e-08, 1.2525339876390262e-08, 3.4802988579940575e-07, 1.9591639954796847e-07, 1.2404500182583433e-07, 1.2346009725661133e-07, 1.92011366806355e-08, 1.9895020386684337e-07, 3.7876212388709973e-08, 1.0629652713589621e-08, 1.2436973229057458e-08, 2.186162078032794e-07, 2.618108396745811e-07, 1.1128912547064829e-07, 1.0776497560982534e-07, 6.380246109216614e-09, 3.895029632161595e-09, 1.576254788915321e-07, 2.9089968212048234e-09, 1.03809227880447e-08, 2.8134863327977655e-08, 1.0777869263733919e-08, 1.3605805371241786e-08, 2.9236161935841665e-08, 1.918957508451058e-07, 2.1994748067299952e-07, 1. [e2e-predictor] 3263833587018325e-08, 4.900423533626963e-08, 2.9808058243929736e-09, 8.925934658066126e-09, 1.0996840771326788e-08, 7.715520666806697e-09, 7.454385553273823e-09, 5.0860990086221136e-08, 1.5129604946650943e-07, 1.1959995660504319e-08, 1.1323217030678734e-08, 9.39119804144184e-09, 9.585582461113518e-08, 1.9055129030220996e-08, 1.8105044574667772e-08, 6.179151057494892e-08, 6.389322493305372e-08, 1.1853056314237165e-08, 9.372677745034252e-09, 1.2332006349424773e-07, 1.6521848777983905e-08, 1.246099685658919e-07, 4.196111902388111e-09, 3.9965367903960214e-08, 1.255440107428285e-08, 1.4302015571843185e-08, 6.631698745707126e-09, 5.964162053118116e-09, 5.556878956980427e-09, 1.419230528654225e-07, 1.7612896030527736e-08, 3.3801444487835397e-07, 7.856438344333583e-08, 2.9667463152804885e-08, 2.8991748877160717e-06, 1.3787196166958893e-06, 5.313577133847502e-09, 2.5128217728820346e-08, 8.406522056247923e-08, 4.492775573794461e-08, 5.4087468726038423e-08, 2.4238921980668238e-08, 4.0167356019082945e-07, 4.1083040969169815e-08, 5.415281023601892e-08, 8.640659743264223e-09, 5.773174294176897e-08, 2.6442751277500065e-07, 8.953106203080097e-07, 2.799365539374321e-08, 5.889395637836969e-09, 4.178830081968954e-08, 2.888028660663622e-08, 2.1357223545237503e-08, 2.3023956430279213e-07, 8.786517469161481e-08, 2.069703963769598e-09, 2.2363829899063603e-08, 3.2032425600903025e-09, 1.176860120466472e-08, 6.963498577761129e-08, 2.2719042647167953e-08, 7.360319465021803e-09, 6.922469264480924e-09, 3.2138252947788715e-08, 1.3705631829452614e-07, 1.9814805796158907e-08, 1.0578844467090676e-08, 2.7049035011827982e-08, 2.9755513608620277e-09, 2.1773475467057324e-08, 1.0975404585167325e-08, 1.9918440230526357e-08, 2.3881801425318372e-08, 2.1357100976615584e-08, 6.109046157831699e-09, 1.1890372242362446e-08, 1.1459750481890296e-08, 3.731681275098708e-08, 1.5726065782928345e-08, 3.403987491878979e-08, 3.6921193213856895e-08, 9.281668411631472e-08, 1.2323060616381554e-07, 4.234707162709128e-08, 1.7423577247654976e-08, 5.811278214196136e-08, 3.9313878374969136e-08, 2.3690166273127033e-08, 1.792834680713895e-08, 1.4406491999352511e-08, 7.0197305745978156e-09, 6.041436506620812e-08, 4.867428771149207e-08, 1.0685186957459791e-08, 1.005112082452797e-08, 4.2425732260653604e-08, 2.6077879411445792e-08, 5.136626679558276e-09, 1.6972753202892932e-09, 1.9131350725842822e-08, 2.1117190840413969e-07, 1.3372530638378066e-08, 2.000219723186092e-08, 1.0454146348592985e-07, 2.814382682458927e-08, 2.134452614654947e-07, 2.104584950757271e-08, 1.9114216343041335e-07, 3.957645233754192e-08, 2.9315843974586642e-08, 1.1051893977764848e-07, 4.840025980001883e-08, 5.583537188158516e-08, 1.2129960680340446e-07, 1.7761975001917563e-08, 2.561053769056798e-08, 1.2038481145282276e-07, 4.6748018789344314e-09, 1.591587306393194e-08, 3.147561500327356e-08, 6.147622144681009e-08, 2.204614979461894e-08, 3.2577264619249036e-07, 1.198899752807847e-07, 2.3818718375423487e-07, 1.490982981522393e-08, 5.1016343149967724e-08, 5.5141523347401744e-08, 2.288526701477167e-08, 5.7140397302646306e-08, 5.185041231925425e-07, 4.977233558633998e-08, 1.104976110610778e-08, 1.26408238543263e-07, 8.174780674607973e-08],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_paddle.py::test_paddle_v2_kserve [e2e-predictor] predictor/test_paddle.py::test_paddle_v2_grpc [e2e-predictor] [gw0] SKIPPED predictor/test_paddle.py::test_paddle_v2_grpc [e2e-predictor] predictor/test_pmml.py::test_pmml_kserve 2026-07-28 17:27:19.964 4494 kserve.trace Host from isvc status URL = isvc-pmml-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:27:19.964 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-pmml-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:27:19.965 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:27:19.965 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:27:19.965 4494 kserve.trace Using external route host: isvc-pmml-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:27:19.965 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-pmml-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:27:19.978 4494 kserve.trace Sending Header = {'Host': 'isvc-pmml-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:27:19.978 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-pmml-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:27:19.979 4494 kserve.trace base url = https://isvc-pmml-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:27:19.979 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-pmml-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:27:20.835 4494 kserve.trace Got response {'predictions': [{'Species': 'setosa', 'Probability_setosa': 1.0, 'Probability_versicolor': 0.0, 'Probability_virginica': 0.0, 'Node_Id': '2'}]} [e2e-predictor] 2026-07-28 17:27:20.835 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [{'Species': 'setosa', 'Probability_setosa': 1.0, 'Probability_versicolor': 0.0, 'Probability_virginica': 0.0, 'Node_Id': '2'}]} [e2e-predictor] 2026-07-28 17:27:20.836 4494 kserve.trace Time taken: 0.9168594679977105 [e2e-predictor] 2026-07-28 17:27:20.836 4494 kserve.trace INFO [test_pmml.py:test_pmml_kserve():77] Time taken: 0.9168594679977105 [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_pmml.py::test_pmml_kserve [e2e-predictor] predictor/test_pmml.py::test_pmml_runtime_kserve 2026-07-28 17:28:56.104 4494 kserve.trace Host from isvc status URL = isvc-pmml-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:28:56.104 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-pmml-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:28:56.104 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:28:56.104 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:28:56.105 4494 kserve.trace Using external route host: isvc-pmml-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:28:56.105 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-pmml-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:28:56.105 4494 kserve.trace Sending Header = {'Host': 'isvc-pmml-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:28:56.105 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-pmml-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:28:56.105 4494 kserve.trace base url = https://isvc-pmml-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:28:56.105 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-pmml-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:28:57.183 4494 kserve.trace Got response {'predictions': [{'Species': 'setosa', 'Probability_setosa': 1.0, 'Probability_versicolor': 0.0, 'Probability_virginica': 0.0, 'Node_Id': '2'}]} [e2e-predictor] 2026-07-28 17:28:57.183 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [{'Species': 'setosa', 'Probability_setosa': 1.0, 'Probability_versicolor': 0.0, 'Probability_virginica': 0.0, 'Node_Id': '2'}]} [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_pmml.py::test_pmml_runtime_kserve [e2e-predictor] predictor/test_pmml.py::test_pmml_v2_kserve 2026-07-28 17:34:54.063 4494 kserve.trace Host from isvc status URL = isvc-pmml-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:34:54.063 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-pmml-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:34:54.064 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:34:54.064 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:34:54.064 4494 kserve.trace Using external route host: isvc-pmml-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:34:54.064 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-pmml-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:34:54.065 4494 kserve.trace Sending Header = {'Host': 'isvc-pmml-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:34:54.065 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-pmml-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:34:54.065 4494 kserve.trace base url = https://isvc-pmml-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:34:54.065 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-pmml-v2-kserve-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:34:54.989 4494 kserve.trace Got response "id": "1bd17ebc-7716-4979-95e5-34d20c2b6770","model_name": "isvc-pmml-v2-kserve","outputs": ["name": "Species","shape": [1],"datatype": "BYTES","data": ['setosa'],"parameters": None, "name": "Probability_setosa","shape": [1],"datatype": "FP64","data": [1.0],"parameters": None, "name": "Probability_versicolor","shape": [1],"datatype": "FP64","data": [0.0],"parameters": None, "name": "Probability_virginica","shape": [1],"datatype": "FP64","data": [0.0],"parameters": None, "name": "Node_Id","shape": [1],"datatype": "BYTES","data": ['2'],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] 2026-07-28 17:34:54.989 4494 kserve.trace INFO [utils.py:predict():148] Got response "id": "1bd17ebc-7716-4979-95e5-34d20c2b6770","model_name": "isvc-pmml-v2-kserve","outputs": ["name": "Species","shape": [1],"datatype": "BYTES","data": ['setosa'],"parameters": None, "name": "Probability_setosa","shape": [1],"datatype": "FP64","data": [1.0],"parameters": None, "name": "Probability_versicolor","shape": [1],"datatype": "FP64","data": [0.0],"parameters": None, "name": "Probability_virginica","shape": [1],"datatype": "FP64","data": [0.0],"parameters": None, "name": "Node_Id","shape": [1],"datatype": "BYTES","data": ['2'],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_pmml.py::test_pmml_v2_kserve [e2e-predictor] predictor/test_pmml.py::test_pmml_v2_grpc [e2e-predictor] [gw0] SKIPPED predictor/test_pmml.py::test_pmml_v2_grpc [e2e-predictor] predictor/test_pod_watch.py::test_event_storm_prevention_init_container_isolation 2026-07-28 17:35:00.162 4494 kserve.trace Created primary ISVC: isvc-primary-bff273 [e2e-predictor] 2026-07-28 17:35:00.162 4494 kserve.trace INFO [test_pod_watch.py:test_event_storm_prevention_init_container_isolation():384] Created primary ISVC: isvc-primary-bff273 [e2e-predictor] 2026-07-28 17:36:30.213 4494 kserve.trace Primary ISVC is ready [e2e-predictor] 2026-07-28 17:36:30.213 4494 kserve.trace INFO [test_pod_watch.py:test_event_storm_prevention_init_container_isolation():386] Primary ISVC is ready [e2e-predictor] 2026-07-28 17:36:45.247 4494 kserve.trace resourceVersion for isvc-primary-bff273 stabilized at 51940 (stable for 15.0s) [e2e-predictor] 2026-07-28 17:36:45.247 4494 kserve.trace INFO [test_pod_watch.py:wait_for_stable_resource_version():185] resourceVersion for isvc-primary-bff273 stabilized at 51940 (stable for 15.0s) [e2e-predictor] 2026-07-28 17:36:45.247 4494 kserve.trace Baseline recorded - resourceVersion: 51940 [e2e-predictor] 2026-07-28 17:36:45.247 4494 kserve.trace INFO [test_pod_watch.py:test_event_storm_prevention_init_container_isolation():394] Baseline recorded - resourceVersion: 51940 [e2e-predictor] 2026-07-28 17:36:45.247 4494 kserve.trace Creating invalid S3 secret and service account [e2e-predictor] 2026-07-28 17:36:45.247 4494 kserve.trace INFO [test_pod_watch.py:test_event_storm_prevention_init_container_isolation():397] Creating invalid S3 secret and service account [e2e-predictor] 2026-07-28 17:36:45.397 4494 kserve.trace Created secondary ISVC isvc-secondary-bff273, waiting for failure status... [e2e-predictor] 2026-07-28 17:36:45.397 4494 kserve.trace INFO [test_pod_watch.py:test_event_storm_prevention_init_container_isolation():427] Created secondary ISVC isvc-secondary-bff273, waiting for failure status... [e2e-predictor] 2026-07-28 17:36:51.416 4494 kserve.trace ISVC isvc-secondary-bff273 reported failure: {'exitCode': 1, 'message': "2026-07-28 17:36:46.357 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('s3://nonexistent-bucket-12345/invalid/path/model', '/mnt/models')]\n2026-07-28 17:36:46.357 1 storage.initializer INFO [kserve_storage.py:download():168] Copying contents of s3://nonexistent-bucket-12345/invalid/path/model to local\n2026-07-28 17:36:46.376 1 storage.initializer INFO [kserve_storage.py:_configure_global_ca_bundle():449] Global ca bundle file(/etc/ssl/custom-certs/cabundle.crt) combined with the default trust store at /tmp/tmpzjnbeeia.crt\n2026-07-28 17:36:46.649 1 storage.initializer INFO [kserve_storage.py:_get_s3_client_kwargs():375] ca bundle file(/etc/ssl/custom-certs/cabundle.crt) exists.\n2026-07-28 17:36:50.672 1 storage.initializer ERROR [initializer-entrypoint:():31] Unexpected error during storage initialization: SSL validation failed for https://s3.amazonaws.com/nonexistent-bucket-12345 [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1004)\n", 'reason': 'ModelLoadFailed'} [e2e-predictor] 2026-07-28 17:36:51.416 4494 kserve.trace INFO [test_pod_watch.py:wait_for_isvc_failure_status():293] ISVC isvc-secondary-bff273 reported failure: {'exitCode': 1, 'message': "2026-07-28 17:36:46.357 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('s3://nonexistent-bucket-12345/invalid/path/model', '/mnt/models')]\n2026-07-28 17:36:46.357 1 storage.initializer INFO [kserve_storage.py:download():168] Copying contents of s3://nonexistent-bucket-12345/invalid/path/model to local\n2026-07-28 17:36:46.376 1 storage.initializer INFO [kserve_storage.py:_configure_global_ca_bundle():449] Global ca bundle file(/etc/ssl/custom-certs/cabundle.crt) combined with the default trust store at /tmp/tmpzjnbeeia.crt\n2026-07-28 17:36:46.649 1 storage.initializer INFO [kserve_storage.py:_get_s3_client_kwargs():375] ca bundle file(/etc/ssl/custom-certs/cabundle.crt) exists.\n2026-07-28 17:36:50.672 1 storage.initializer ERROR [initializer-entrypoint:():31] Unexpected error during storage initialization: SSL validation failed for https://s3.amazonaws.com/nonexistent-bucket-12345 [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1004)\n", 'reason': 'ModelLoadFailed'} [e2e-predictor] 2026-07-28 17:36:51.417 4494 kserve.trace Secondary ISVC failure detected: {'copies': {'failedCopies': 0}, 'lastFailureInfo': {'exitCode': 1, 'message': "2026-07-28 17:36:46.357 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('s3://nonexistent-bucket-12345/invalid/path/model', '/mnt/models')]\n2026-07-28 17:36:46.357 1 storage.initializer INFO [kserve_storage.py:download():168] Copying contents of s3://nonexistent-bucket-12345/invalid/path/model to local\n2026-07-28 17:36:46.376 1 storage.initializer INFO [kserve_storage.py:_configure_global_ca_bundle():449] Global ca bundle file(/etc/ssl/custom-certs/cabundle.crt) combined with the default trust store at /tmp/tmpzjnbeeia.crt\n2026-07-28 17:36:46.649 1 storage.initializer INFO [kserve_storage.py:_get_s3_client_kwargs():375] ca bundle file(/etc/ssl/custom-certs/cabundle.crt) exists.\n2026-07-28 17:36:50.672 1 storage.initializer ERROR [initializer-entrypoint:():31] Unexpected error during storage initialization: SSL validation failed for https://s3.amazonaws.com/nonexistent-bucket-12345 [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1004)\n", 'reason': 'ModelLoadFailed'}, 'states': {'activeModelState': '', 'targetModelState': 'FailedToLoad'}, 'transitionStatus': 'BlockedByFailedLoad'} [e2e-predictor] 2026-07-28 17:36:51.417 4494 kserve.trace INFO [test_pod_watch.py:test_event_storm_prevention_init_container_isolation():435] Secondary ISVC failure detected: {'copies': {'failedCopies': 0}, 'lastFailureInfo': {'exitCode': 1, 'message': "2026-07-28 17:36:46.357 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('s3://nonexistent-bucket-12345/invalid/path/model', '/mnt/models')]\n2026-07-28 17:36:46.357 1 storage.initializer INFO [kserve_storage.py:download():168] Copying contents of s3://nonexistent-bucket-12345/invalid/path/model to local\n2026-07-28 17:36:46.376 1 storage.initializer INFO [kserve_storage.py:_configure_global_ca_bundle():449] Global ca bundle file(/etc/ssl/custom-certs/cabundle.crt) combined with the default trust store at /tmp/tmpzjnbeeia.crt\n2026-07-28 17:36:46.649 1 storage.initializer INFO [kserve_storage.py:_get_s3_client_kwargs():375] ca bundle file(/etc/ssl/custom-certs/cabundle.crt) exists.\n2026-07-28 17:36:50.672 1 storage.initializer ERROR [initializer-entrypoint:():31] Unexpected error during storage initialization: SSL validation failed for https://s3.amazonaws.com/nonexistent-bucket-12345 [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1004)\n", 'reason': 'ModelLoadFailed'}, 'states': {'activeModelState': '', 'targetModelState': 'FailedToLoad'}, 'transitionStatus': 'BlockedByFailedLoad'} [e2e-predictor] 2026-07-28 17:37:01.432 4494 kserve.trace Primary ISVC resourceVersion: before=51940, after=51940 [e2e-predictor] 2026-07-28 17:37:01.432 4494 kserve.trace INFO [test_pod_watch.py:test_event_storm_prevention_init_container_isolation():445] Primary ISVC resourceVersion: before=51940, after=51940 [e2e-predictor] 2026-07-28 17:37:01.432 4494 kserve.trace Event storm prevention validated: Primary ISVC was not modified during secondary ISVC init container failures [e2e-predictor] 2026-07-28 17:37:01.432 4494 kserve.trace INFO [test_pod_watch.py:test_event_storm_prevention_init_container_isolation():458] Event storm prevention validated: Primary ISVC was not modified during secondary ISVC init container failures [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_pod_watch.py::test_event_storm_prevention_init_container_isolation [e2e-predictor] predictor/test_pod_watch.py::test_quick_reconciliation_on_init_container_failure 2026-07-28 17:37:06.571 4494 kserve.trace Creating invalid S3 secret and service account [e2e-predictor] 2026-07-28 17:37:06.571 4494 kserve.trace INFO [test_pod_watch.py:test_quick_reconciliation_on_init_container_failure():489] Creating invalid S3 secret and service account [e2e-predictor] 2026-07-28 17:37:06.705 4494 kserve.trace Created ISVC isvc-init-fail-302ca9, waiting for failure status... [e2e-predictor] 2026-07-28 17:37:06.705 4494 kserve.trace INFO [test_pod_watch.py:test_quick_reconciliation_on_init_container_failure():520] Created ISVC isvc-init-fail-302ca9, waiting for failure status... [e2e-predictor] 2026-07-28 17:37:11.715 4494 kserve.trace ISVC isvc-init-fail-302ca9 reported failure: {'exitCode': 1, 'message': "2026-07-28 17:37:08.189 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('s3://nonexistent-bucket-xyz123/invalid/model', '/mnt/models')]\n2026-07-28 17:37:08.190 1 storage.initializer INFO [kserve_storage.py:download():168] Copying contents of s3://nonexistent-bucket-xyz123/invalid/model to local\n2026-07-28 17:37:08.208 1 storage.initializer INFO [kserve_storage.py:_configure_global_ca_bundle():449] Global ca bundle file(/etc/ssl/custom-certs/cabundle.crt) combined with the default trust store at /tmp/tmpwbxqivqu.crt\n2026-07-28 17:37:08.482 1 storage.initializer INFO [kserve_storage.py:_get_s3_client_kwargs():375] ca bundle file(/etc/ssl/custom-certs/cabundle.crt) exists.\n2026-07-28 17:37:09.694 1 storage.initializer ERROR [initializer-entrypoint:():31] Unexpected error during storage initialization: SSL validation failed for https://s3.amazonaws.com/nonexistent-bucket-xyz123 [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1004)\n", 'reason': 'ModelLoadFailed'} [e2e-predictor] 2026-07-28 17:37:11.715 4494 kserve.trace INFO [test_pod_watch.py:wait_for_isvc_failure_status():293] ISVC isvc-init-fail-302ca9 reported failure: {'exitCode': 1, 'message': "2026-07-28 17:37:08.189 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('s3://nonexistent-bucket-xyz123/invalid/model', '/mnt/models')]\n2026-07-28 17:37:08.190 1 storage.initializer INFO [kserve_storage.py:download():168] Copying contents of s3://nonexistent-bucket-xyz123/invalid/model to local\n2026-07-28 17:37:08.208 1 storage.initializer INFO [kserve_storage.py:_configure_global_ca_bundle():449] Global ca bundle file(/etc/ssl/custom-certs/cabundle.crt) combined with the default trust store at /tmp/tmpwbxqivqu.crt\n2026-07-28 17:37:08.482 1 storage.initializer INFO [kserve_storage.py:_get_s3_client_kwargs():375] ca bundle file(/etc/ssl/custom-certs/cabundle.crt) exists.\n2026-07-28 17:37:09.694 1 storage.initializer ERROR [initializer-entrypoint:():31] Unexpected error during storage initialization: SSL validation failed for https://s3.amazonaws.com/nonexistent-bucket-xyz123 [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1004)\n", 'reason': 'ModelLoadFailed'} [e2e-predictor] 2026-07-28 17:37:11.715 4494 kserve.trace Failure status detected in 5.06 seconds: {'copies': {'failedCopies': 0}, 'lastFailureInfo': {'exitCode': 1, 'message': "2026-07-28 17:37:08.189 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('s3://nonexistent-bucket-xyz123/invalid/model', '/mnt/models')]\n2026-07-28 17:37:08.190 1 storage.initializer INFO [kserve_storage.py:download():168] Copying contents of s3://nonexistent-bucket-xyz123/invalid/model to local\n2026-07-28 17:37:08.208 1 storage.initializer INFO [kserve_storage.py:_configure_global_ca_bundle():449] Global ca bundle file(/etc/ssl/custom-certs/cabundle.crt) combined with the default trust store at /tmp/tmpwbxqivqu.crt\n2026-07-28 17:37:08.482 1 storage.initializer INFO [kserve_storage.py:_get_s3_client_kwargs():375] ca bundle file(/etc/ssl/custom-certs/cabundle.crt) exists.\n2026-07-28 17:37:09.694 1 storage.initializer ERROR [initializer-entrypoint:():31] Unexpected error during storage initialization: SSL validation failed for https://s3.amazonaws.com/nonexistent-bucket-xyz123 [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1004)\n", 'reason': 'ModelLoadFailed'}, 'states': {'activeModelState': '', 'targetModelState': 'FailedToLoad'}, 'transitionStatus': 'BlockedByFailedLoad'} [e2e-predictor] 2026-07-28 17:37:11.715 4494 kserve.trace INFO [test_pod_watch.py:test_quick_reconciliation_on_init_container_failure():534] Failure status detected in 5.06 seconds: {'copies': {'failedCopies': 0}, 'lastFailureInfo': {'exitCode': 1, 'message': "2026-07-28 17:37:08.189 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('s3://nonexistent-bucket-xyz123/invalid/model', '/mnt/models')]\n2026-07-28 17:37:08.190 1 storage.initializer INFO [kserve_storage.py:download():168] Copying contents of s3://nonexistent-bucket-xyz123/invalid/model to local\n2026-07-28 17:37:08.208 1 storage.initializer INFO [kserve_storage.py:_configure_global_ca_bundle():449] Global ca bundle file(/etc/ssl/custom-certs/cabundle.crt) combined with the default trust store at /tmp/tmpwbxqivqu.crt\n2026-07-28 17:37:08.482 1 storage.initializer INFO [kserve_storage.py:_get_s3_client_kwargs():375] ca bundle file(/etc/ssl/custom-certs/cabundle.crt) exists.\n2026-07-28 17:37:09.694 1 storage.initializer ERROR [initializer-entrypoint:():31] Unexpected error during storage initialization: SSL validation failed for https://s3.amazonaws.com/nonexistent-bucket-xyz123 [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1004)\n", 'reason': 'ModelLoadFailed'}, 'states': {'activeModelState': '', 'targetModelState': 'FailedToLoad'}, 'transitionStatus': 'BlockedByFailedLoad'} [e2e-predictor] 2026-07-28 17:37:11.715 4494 kserve.trace Transition status: BlockedByFailedLoad [e2e-predictor] 2026-07-28 17:37:11.715 4494 kserve.trace INFO [test_pod_watch.py:test_quick_reconciliation_on_init_container_failure():548] Transition status: BlockedByFailedLoad [e2e-predictor] 2026-07-28 17:37:11.720 4494 kserve.trace Ready condition: {'lastTransitionTime': '2026-07-28T17:37:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'} [e2e-predictor] 2026-07-28 17:37:11.720 4494 kserve.trace INFO [test_pod_watch.py:test_quick_reconciliation_on_init_container_failure():557] Ready condition: {'lastTransitionTime': '2026-07-28T17:37:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'} [e2e-predictor] 2026-07-28 17:37:11.720 4494 kserve.trace Quick reconciliation validated: Failure detected in 5.06 seconds [e2e-predictor] 2026-07-28 17:37:11.720 4494 kserve.trace INFO [test_pod_watch.py:test_quick_reconciliation_on_init_container_failure():570] Quick reconciliation validated: Failure detected in 5.06 seconds [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_pod_watch.py::test_quick_reconciliation_on_init_container_failure [e2e-predictor] predictor/test_predictive.py::test_predictive_sklearn_v1 2026-07-28 17:39:07.054 4494 kserve.trace Host from isvc status URL = isvc-predictive-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:39:07.054 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-predictive-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:39:07.054 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:39:07.054 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:39:07.055 4494 kserve.trace Using external route host: isvc-predictive-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:39:07.055 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-predictive-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:39:07.055 4494 kserve.trace Sending Header = {'Host': 'isvc-predictive-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:39:07.055 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-predictive-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:39:07.055 4494 kserve.trace base url = https://isvc-predictive-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:39:07.055 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-predictive-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:39:07.091 4494 kserve.trace Got response {'predictions': [1, 1]} [e2e-predictor] 2026-07-28 17:39:07.091 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [1, 1]} [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_predictive.py::test_predictive_sklearn_v1 [e2e-predictor] predictor/test_predictive.py::test_predictive_xgboost_v1 2026-07-28 17:40:42.377 4494 kserve.trace Host from isvc status URL = isvc-predictive-xgboost-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:40:42.377 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-predictive-xgboost-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:40:42.377 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:40:42.377 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:40:42.378 4494 kserve.trace Using external route host: isvc-predictive-xgboost-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:40:42.378 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-predictive-xgboost-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:40:42.378 4494 kserve.trace Sending Header = {'Host': 'isvc-predictive-xgboost-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:40:42.378 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-predictive-xgboost-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:40:42.378 4494 kserve.trace base url = https://isvc-predictive-xgboost-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:40:42.378 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-predictive-xgboost-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:40:42.414 4494 kserve.trace Got response {'predictions': [1.0, 1.0]} [e2e-predictor] 2026-07-28 17:40:42.414 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [1.0, 1.0]} [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_predictive.py::test_predictive_xgboost_v1 [e2e-predictor] predictor/test_predictive.py::test_predictive_lightgbm_v1 2026-07-28 17:42:17.682 4494 kserve.trace Host from isvc status URL = isvc-predictive-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:42:17.682 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-predictive-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:42:17.683 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:42:17.683 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:42:17.683 4494 kserve.trace Using external route host: isvc-predictive-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:42:17.683 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-predictive-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:42:17.683 4494 kserve.trace Sending Header = {'Host': 'isvc-predictive-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:42:17.683 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-predictive-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:42:17.683 4494 kserve.trace base url = https://isvc-predictive-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:42:17.683 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-predictive-lightgbm-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:42:17.718 4494 kserve.trace Got response {'predictions': [[0.9999378629898321, 4.415799218835629e-05, 1.797901797954119e-05]]} [e2e-predictor] 2026-07-28 17:42:17.718 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [[0.9999378629898321, 4.415799218835629e-05, 1.797901797954119e-05]]} [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_predictive.py::test_predictive_lightgbm_v1 [e2e-predictor] predictor/test_predictive.py::test_predictive_sklearn_v2 2026-07-28 17:43:42.990 4494 kserve.trace Host from isvc status URL = isvc-predictive-sklearn-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:43:42.990 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-predictive-sklearn-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:43:42.990 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:43:42.990 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:43:42.990 4494 kserve.trace Using external route host: isvc-predictive-sklearn-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:43:42.990 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-predictive-sklearn-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:43:42.990 4494 kserve.trace Sending Header = {'Host': 'isvc-predictive-sklearn-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:43:42.990 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-predictive-sklearn-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:43:42.990 4494 kserve.trace base url = https://isvc-predictive-sklearn-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:43:42.990 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-predictive-sklearn-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:43:43.885 4494 kserve.trace Got response "id": "25d53ccd-98d0-4b9a-81f8-714079ff4c49","model_name": "isvc-predictive-sklearn-v2","outputs": ["name": "output-0","shape": [2],"datatype": "INT32","data": [1, 1],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] 2026-07-28 17:43:43.885 4494 kserve.trace INFO [utils.py:predict():148] Got response "id": "25d53ccd-98d0-4b9a-81f8-714079ff4c49","model_name": "isvc-predictive-sklearn-v2","outputs": ["name": "output-0","shape": [2],"datatype": "INT32","data": [1, 1],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_predictive.py::test_predictive_sklearn_v2 [e2e-predictor] predictor/test_predictive.py::test_predictive_xgboost_v2 2026-07-28 17:45:09.168 4494 kserve.trace Host from isvc status URL = isvc-predictive-xgboost-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:45:09.168 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-predictive-xgboost-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:45:09.169 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:45:09.169 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:45:09.169 4494 kserve.trace Using external route host: isvc-predictive-xgboost-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:45:09.169 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-predictive-xgboost-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:45:09.169 4494 kserve.trace Sending Header = {'Host': 'isvc-predictive-xgboost-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:45:09.169 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-predictive-xgboost-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:45:09.169 4494 kserve.trace base url = https://isvc-predictive-xgboost-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:45:09.169 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-predictive-xgboost-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:45:09.200 4494 kserve.trace Got response "id": "4bb1fa5b-2070-42fe-8a5a-bbbb3bb4fd0f","model_name": "isvc-predictive-xgboost-v2","outputs": ["name": "output-0","shape": [2],"datatype": "FP32","data": [1.0, 1.0],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] 2026-07-28 17:45:09.200 4494 kserve.trace INFO [utils.py:predict():148] Got response "id": "4bb1fa5b-2070-42fe-8a5a-bbbb3bb4fd0f","model_name": "isvc-predictive-xgboost-v2","outputs": ["name": "output-0","shape": [2],"datatype": "FP32","data": [1.0, 1.0],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_predictive.py::test_predictive_xgboost_v2 [e2e-predictor] predictor/test_predictive.py::test_predictive_lightgbm_v2 2026-07-28 17:46:44.461 4494 kserve.trace Host from isvc status URL = isvc-predictive-lightgbm-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:46:44.461 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-predictive-lightgbm-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:46:44.461 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:46:44.461 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:46:44.461 4494 kserve.trace Using external route host: isvc-predictive-lightgbm-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:46:44.461 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-predictive-lightgbm-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:46:44.461 4494 kserve.trace Sending Header = {'Host': 'isvc-predictive-lightgbm-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:46:44.461 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-predictive-lightgbm-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:46:44.461 4494 kserve.trace base url = https://isvc-predictive-lightgbm-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:46:44.461 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-predictive-lightgbm-v2-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:46:44.489 4494 kserve.trace Got response "id": "1c298f93-481c-495d-b5dc-0a4ce0eb8428","model_name": "isvc-predictive-lightgbm-v2","outputs": ["name": "output-0","shape": [2, 3],"datatype": "FP64","data": [0.00028762259763255566, 0.9763481320224064, 0.023364245379961103, 0.0005625431624697456, 0.9988642605470336, 0.0005731962904966214],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] 2026-07-28 17:46:44.489 4494 kserve.trace INFO [utils.py:predict():148] Got response "id": "1c298f93-481c-495d-b5dc-0a4ce0eb8428","model_name": "isvc-predictive-lightgbm-v2","outputs": ["name": "output-0","shape": [2, 3],"datatype": "FP64","data": [0.00028762259763255566, 0.9763481320224064, 0.023364245379961103, 0.0005625431624697456, 0.9988642605470336, 0.0005731962904966214],"parameters": None],"parameters": None,"from_grpc": False [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_predictive.py::test_predictive_lightgbm_v2 [e2e-predictor] predictor/test_scheduler_name.py::test_scheduler_name 2026-07-28T17:48:58.859140+00:00 timeout_sampler INFO Waiting for 30 seconds [0:00:30], retry every 2 seconds. (Function: e2e.predictor.test_scheduler_name..lambda: kserve_client.service_name.get_pods) [e2e-predictor] 2026-07-28T17:49:00.906758+00:00 timeout_sampler INFO Elapsed time: 2.036273717880249 [0:00:02.036274] [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_scheduler_name.py::test_scheduler_name [e2e-predictor] predictor/test_sklearn.py::test_sklearn_kserve 2026-07-28 17:50:36.169 4494 kserve.trace Host from isvc status URL = isvc-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:50:36.169 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:50:36.169 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:50:36.169 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:50:36.170 4494 kserve.trace Using external route host: isvc-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:50:36.170 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:50:36.170 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:50:36.170 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:50:36.170 4494 kserve.trace base url = https://isvc-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:50:36.170 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:50:36.198 4494 kserve.trace Got response {'predictions': [1, 1]} [e2e-predictor] 2026-07-28 17:50:36.198 4494 kserve.trace INFO [utils.py:predict():148] Got response {'predictions': [1, 1]} [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_sklearn.py::test_sklearn_kserve [e2e-predictor] predictor/test_sklearn.py::test_sklearn_v2_mlserver 2026-07-28 17:51:41.460 4494 kserve.trace Host from isvc status URL = sklearn-v2-mlserver-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:51:41.460 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = sklearn-v2-mlserver-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:51:41.461 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:51:41.461 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:51:41.461 4494 kserve.trace Using external route host: sklearn-v2-mlserver-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:51:41.461 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: sklearn-v2-mlserver-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:51:41.461 4494 kserve.trace Sending Header = {'Host': 'sklearn-v2-mlserver-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:51:41.461 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'sklearn-v2-mlserver-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:51:41.461 4494 kserve.trace base url = https://sklearn-v2-mlserver-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:51:41.461 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://sklearn-v2-mlserver-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:51:41.493 4494 kserve.trace Got response "id": "88e1cf36-4443-4ee4-9fb7-28dd771e658c","model_name": "sklearn-v2-mlserver","outputs": ["name": "predict","shape": [2, 1],"datatype": "INT32","data": [1, 1],"parameters": {'content_type': 'np'}],"parameters": {},"from_grpc": False [e2e-predictor] 2026-07-28 17:51:41.493 4494 kserve.trace INFO [utils.py:predict():148] Got response "id": "88e1cf36-4443-4ee4-9fb7-28dd771e658c","model_name": "sklearn-v2-mlserver","outputs": ["name": "predict","shape": [2, 1],"datatype": "INT32","data": [1, 1],"parameters": {'content_type': 'np'}],"parameters": {},"from_grpc": False [e2e-predictor] [e2e-predictor] [gw0] PASSED predictor/test_sklearn.py::test_sklearn_v2_mlserver [e2e-predictor] predictor/test_sklearn.py::test_sklearn_runtime_kserve 2026-07-28 17:52:16.736 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:52:16.736 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:52:16.737 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:52:16.737 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:52:16.737 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:52:16.737 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:52:16.737 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:52:16.737 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:52:16.738 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:52:16.738 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:52:16.783 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:52:16.783 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:52:16.783 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:52:16.783 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:52:16.783 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:52:16.783 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:52:16.784 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:52:16.784 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:52:16.784 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:52:16.784 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:54:25.472 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:54:25.472 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:54:25.472 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:54:25.472 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:54:25.472 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:54:25.472 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:54:25.473 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:54:25.473 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:54:25.473 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:54:25.473 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.544 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.544 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.545 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.545 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.545 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.545 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.545 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.545 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.545 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.545 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.591 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.591 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.591 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.591 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.591 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.591 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.591 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.591 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.592 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.592 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.648 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.648 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.648 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.648 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.648 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.648 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.648 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.648 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.648 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.648 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.687 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.687 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.687 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.687 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.687 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.687 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.687 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.687 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.687 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.687 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.722 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.722 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.722 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.722 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.722 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.722 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.722 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.722 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.722 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.722 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.764 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.764 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.764 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.764 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.764 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.764 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.764 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.764 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.764 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.764 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.802 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.802 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.802 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.802 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.803 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.803 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.803 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.803 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.803 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.803 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.843 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.843 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.843 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.843 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.843 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.843 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.843 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.843 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.843 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.843 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.882 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.882 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.882 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.882 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.883 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.883 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.883 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.883 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.883 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.883 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.919 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.919 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.920 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.920 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.920 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.920 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.920 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.920 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.920 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.920 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.960 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.960 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.960 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.960 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:56:36.960 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.960 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.960 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.960 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:56:36.960 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:56:36.960 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.617 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.617 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.617 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.617 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.617 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.617 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.617 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.617 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.617 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.617 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.668 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.668 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.668 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.668 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.668 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.668 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.668 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.668 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.668 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.668 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.708 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.708 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.709 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.709 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.709 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.709 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.709 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.709 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.709 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.709 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.752 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.752 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.753 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.753 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.753 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.753 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.753 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.753 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.753 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.753 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.798 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.798 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.798 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.798 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.798 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.798 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.798 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.798 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.798 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.798 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.845 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.845 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.845 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.845 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.845 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.845 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.846 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.846 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.846 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.846 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.881 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.881 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.881 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.881 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.882 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.882 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.882 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.882 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.882 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.882 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.919 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.919 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.919 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.919 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 17:58:47.919 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.919 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.919 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.919 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 17:58:47.919 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 17:58:47.919 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.688 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.688 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.689 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 18:00:58.689 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 18:00:58.689 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.689 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.689 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 18:00:58.689 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 18:00:58.689 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.689 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.731 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.731 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.731 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 18:00:58.731 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 18:00:58.731 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.731 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.731 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 18:00:58.731 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 18:00:58.732 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.732 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.783 4494 kserve.trace Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.783 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():373] Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.783 4494 kserve.trace Network layer = openshift-route [e2e-predictor] 2026-07-28 18:00:58.783 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():374] Network layer = openshift-route [e2e-predictor] 2026-07-28 18:00:58.783 4494 kserve.trace Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.783 4494 kserve.trace INFO [utils.py:get_isvc_endpoint():377] Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.783 4494 kserve.trace Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 18:00:58.783 4494 kserve.trace INFO [utils.py:_predict():164] Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] 2026-07-28 18:00:58.783 4494 kserve.trace base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] 2026-07-28 18:00:58.783 4494 kserve.trace INFO [utils.py:_predict():165] base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] [e2e-predictor] [gw0] FAILED predictor/test_sklearn.py::test_sklearn_runtime_kserve [e2e-predictor] predictor/test_sklearn.py::test_sklearn_v2_runtime_mlserver [e2e-predictor] [gw0] ERROR predictor/test_sklearn.py::test_sklearn_v2_runtime_mlserver [e2e-predictor] predictor/test_sklearn.py::test_sklearn_v2 [e2e-predictor] [gw0] ERROR predictor/test_sklearn.py::test_sklearn_v2 [e2e-predictor] predictor/test_sklearn.py::test_sklearn_v2_grpc [e2e-predictor] [gw0] SKIPPED predictor/test_sklearn.py::test_sklearn_v2_grpc [e2e-predictor] predictor/test_sklearn.py::test_sklearn_v2_mixed [e2e-predictor] [gw0] ERROR predictor/test_sklearn.py::test_sklearn_v2_mixed [e2e-predictor] predictor/test_sklearn.py::test_sklearn_v2_mixed_grpc [e2e-predictor] [gw0] SKIPPED predictor/test_sklearn.py::test_sklearn_v2_mixed_grpc [e2e-predictor] predictor/test_tensorflow.py::test_tensorflow_kserve [e2e-predictor] [gw0] ERROR predictor/test_tensorflow.py::test_tensorflow_kserve [e2e-predictor] [e2e-predictor] ==================================== ERRORS ==================================== [e2e-predictor] ______________ ERROR at setup of test_sklearn_v2_runtime_mlserver ______________ [e2e-predictor] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-predictor] [e2e-predictor] self = [e2e-predictor] [e2e-predictor] def _new_conn(self) -> socket.socket: [e2e-predictor] """Establish a socket connection and set nodelay settings on it. [e2e-predictor] [e2e-predictor] :return: New socket connection. [e2e-predictor] """ [e2e-predictor] try: [e2e-predictor] > sock = connection.create_connection( [e2e-predictor] (self._dns_host, self.port), [e2e-predictor] self.timeout, [e2e-predictor] source_address=self.source_address, [e2e-predictor] socket_options=self.socket_options, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:204: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] address = ('a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', 6443) [e2e-predictor] timeout = None, source_address = None, socket_options = [(6, 1, 1)] [e2e-predictor] [e2e-predictor] def create_connection( [e2e-predictor] address: tuple[str, int], [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] source_address: tuple[str, int] | None = None, [e2e-predictor] socket_options: _TYPE_SOCKET_OPTIONS | None = None, [e2e-predictor] ) -> socket.socket: [e2e-predictor] """Connect to *address* and return the socket object. [e2e-predictor] [e2e-predictor] Convenience function. Connect to *address* (a 2-tuple ``(host, [e2e-predictor] port)``) and return the socket object. Passing the optional [e2e-predictor] *timeout* parameter will set the timeout on the socket instance [e2e-predictor] before attempting to connect. If no *timeout* is supplied, the [e2e-predictor] global default timeout setting returned by :func:`socket.getdefaulttimeout` [e2e-predictor] is used. If *source_address* is set it must be a tuple of (host, port) [e2e-predictor] for the socket to bind as a source address before making the connection. [e2e-predictor] An host of '' or port 0 tells the OS to use the default. [e2e-predictor] """ [e2e-predictor] [e2e-predictor] host, port = address [e2e-predictor] if host.startswith("["): [e2e-predictor] host = host.strip("[]") [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Using the value from allowed_gai_family() in the context of getaddrinfo lets [e2e-predictor] # us select whether to work with IPv4 DNS records, IPv6 records, or both. [e2e-predictor] # The original create_connection function always returns all records. [e2e-predictor] family = allowed_gai_family() [e2e-predictor] [e2e-predictor] try: [e2e-predictor] host.encode("idna") [e2e-predictor] except UnicodeError: [e2e-predictor] raise LocationParseError(f"'{host}', label empty or too long") from None [e2e-predictor] [e2e-predictor] > for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/connection.py:60: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] host = 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' [e2e-predictor] port = 6443, family = [e2e-predictor] type = , proto = 0, flags = 0 [e2e-predictor] [e2e-predictor] def getaddrinfo(host, port, family=0, type=0, proto=0, flags=0): [e2e-predictor] """Resolve host and port into list of address info entries. [e2e-predictor] [e2e-predictor] Translate the host/port argument into a sequence of 5-tuples that contain [e2e-predictor] all the necessary arguments for creating a socket connected to that service. [e2e-predictor] host is a domain name, a string representation of an IPv4/v6 address or [e2e-predictor] None. port is a string service name such as 'http', a numeric port number or [e2e-predictor] None. By passing None as the value of host and port, you can pass NULL to [e2e-predictor] the underlying C API. [e2e-predictor] [e2e-predictor] The family, type and proto arguments can be optionally specified in order to [e2e-predictor] narrow the list of addresses returned. Passing zero as a value for each of [e2e-predictor] these arguments selects the full range of results. [e2e-predictor] """ [e2e-predictor] # We override this function since we want to translate the numeric family [e2e-predictor] # and socket type values to enum constants. [e2e-predictor] addrlist = [] [e2e-predictor] > for res in _socket.getaddrinfo(host, port, family, type, proto, flags): [e2e-predictor] E socket.gaierror: [Errno -2] Name or service not known [e2e-predictor] [e2e-predictor] /usr/lib64/python3.11/socket.py:974: gaierror [e2e-predictor] [e2e-predictor] The above exception was the direct cause of the following exception: [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False, err = None [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] > response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:788: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] conn = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] timeout = Timeout(connect=None, read=None, total=None), chunked = False [e2e-predictor] response_conn = None, preload_content = True, decode_content = True [e2e-predictor] enforce_content_length = True [e2e-predictor] [e2e-predictor] def _make_request( [e2e-predictor] self, [e2e-predictor] conn: BaseHTTPConnection, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | None = None, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] chunked: bool = False, [e2e-predictor] response_conn: BaseHTTPConnection | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] enforce_content_length: bool = True, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Perform a request on a given urllib connection object taken from our [e2e-predictor] pool. [e2e-predictor] [e2e-predictor] :param conn: [e2e-predictor] a connection from one of our connection pools [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] Pass ``None`` to retry until you receive a response. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param response_conn: [e2e-predictor] Set this to ``None`` if you will handle releasing the connection or [e2e-predictor] set the connection to have the response release it. [e2e-predictor] [e2e-predictor] :param preload_content: [e2e-predictor] If True, the response's body will be preloaded during construction. [e2e-predictor] [e2e-predictor] :param decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param enforce_content_length: [e2e-predictor] Enforce content length checking. Body returned by server must match [e2e-predictor] value of Content-Length header, if present. Otherwise, raise error. [e2e-predictor] """ [e2e-predictor] self.num_requests += 1 [e2e-predictor] [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] timeout_obj.start_connect() [e2e-predictor] conn.timeout = Timeout.resolve_default_timeout(timeout_obj.connect_timeout) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Trigger any extra validation we need to do. [e2e-predictor] try: [e2e-predictor] self._validate_conn(conn) [e2e-predictor] except (SocketTimeout, BaseSSLError) as e: [e2e-predictor] self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # _validate_conn() starts the connection to an HTTPS proxy [e2e-predictor] # so we need to wrap errors with 'ProxyError' here too. [e2e-predictor] except ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] BaseSSLError, [e2e-predictor] CertificateError, [e2e-predictor] SSLError, [e2e-predictor] ) as e: [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] # If the connection didn't successfully connect to it's proxy [e2e-predictor] # then there [e2e-predictor] if isinstance( [e2e-predictor] new_e, (OSError, NewConnectionError, TimeoutError, SSLError) [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] > raise new_e [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:488: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] conn = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] timeout = Timeout(connect=None, read=None, total=None), chunked = False [e2e-predictor] response_conn = None, preload_content = True, decode_content = True [e2e-predictor] enforce_content_length = True [e2e-predictor] [e2e-predictor] def _make_request( [e2e-predictor] self, [e2e-predictor] conn: BaseHTTPConnection, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | None = None, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] chunked: bool = False, [e2e-predictor] response_conn: BaseHTTPConnection | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] enforce_content_length: bool = True, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Perform a request on a given urllib connection object taken from our [e2e-predictor] pool. [e2e-predictor] [e2e-predictor] :param conn: [e2e-predictor] a connection from one of our connection pools [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] Pass ``None`` to retry until you receive a response. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param response_conn: [e2e-predictor] Set this to ``None`` if you will handle releasing the connection or [e2e-predictor] set the connection to have the response release it. [e2e-predictor] [e2e-predictor] :param preload_content: [e2e-predictor] If True, the response's body will be preloaded during construction. [e2e-predictor] [e2e-predictor] :param decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param enforce_content_length: [e2e-predictor] Enforce content length checking. Body returned by server must match [e2e-predictor] value of Content-Length header, if present. Otherwise, raise error. [e2e-predictor] """ [e2e-predictor] self.num_requests += 1 [e2e-predictor] [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] timeout_obj.start_connect() [e2e-predictor] conn.timeout = Timeout.resolve_default_timeout(timeout_obj.connect_timeout) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Trigger any extra validation we need to do. [e2e-predictor] try: [e2e-predictor] > self._validate_conn(conn) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:464: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] conn = [e2e-predictor] [e2e-predictor] def _validate_conn(self, conn: BaseHTTPConnection) -> None: [e2e-predictor] """ [e2e-predictor] Called right before a request is made, after the socket is created. [e2e-predictor] """ [e2e-predictor] super()._validate_conn(conn) [e2e-predictor] [e2e-predictor] # Force connect early to allow us to validate the connection. [e2e-predictor] if conn.is_closed: [e2e-predictor] > conn.connect() [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:1106: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] [e2e-predictor] def connect(self) -> None: [e2e-predictor] # Today we don't need to be doing this step before the /actual/ socket [e2e-predictor] # connection, however in the future we'll need to decide whether to [e2e-predictor] # create a new socket or re-use an existing "shared" socket as a part [e2e-predictor] # of the HTTP/2 handshake dance. [e2e-predictor] if self._tunnel_host is not None and self._tunnel_port is not None: [e2e-predictor] probe_http2_host = self._tunnel_host [e2e-predictor] probe_http2_port = self._tunnel_port [e2e-predictor] else: [e2e-predictor] probe_http2_host = self.host [e2e-predictor] probe_http2_port = self.port [e2e-predictor] [e2e-predictor] # Check if the target origin supports HTTP/2. [e2e-predictor] # If the value comes back as 'None' it means that the current thread [e2e-predictor] # is probing for HTTP/2 support. Otherwise, we're waiting for another [e2e-predictor] # probe to complete, or we get a value right away. [e2e-predictor] target_supports_http2: bool | None [e2e-predictor] if "h2" in ssl_.ALPN_PROTOCOLS: [e2e-predictor] target_supports_http2 = http2_probe.acquire_and_get( [e2e-predictor] host=probe_http2_host, port=probe_http2_port [e2e-predictor] ) [e2e-predictor] else: [e2e-predictor] # If HTTP/2 isn't going to be offered it doesn't matter if [e2e-predictor] # the target supports HTTP/2. Don't want to make a probe. [e2e-predictor] target_supports_http2 = False [e2e-predictor] [e2e-predictor] if self._connect_callback is not None: [e2e-predictor] self._connect_callback( [e2e-predictor] "before connect", [e2e-predictor] thread_id=threading.get_ident(), [e2e-predictor] target_supports_http2=target_supports_http2, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] sock: socket.socket | ssl.SSLSocket [e2e-predictor] > self.sock = sock = self._new_conn() [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:759: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] [e2e-predictor] def _new_conn(self) -> socket.socket: [e2e-predictor] """Establish a socket connection and set nodelay settings on it. [e2e-predictor] [e2e-predictor] :return: New socket connection. [e2e-predictor] """ [e2e-predictor] try: [e2e-predictor] sock = connection.create_connection( [e2e-predictor] (self._dns_host, self.port), [e2e-predictor] self.timeout, [e2e-predictor] source_address=self.source_address, [e2e-predictor] socket_options=self.socket_options, [e2e-predictor] ) [e2e-predictor] except socket.gaierror as e: [e2e-predictor] > raise NameResolutionError(self.host, self, e) from e [e2e-predictor] E urllib3.exceptions.NameResolutionError: HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:211: NameResolutionError [e2e-predictor] [e2e-predictor] The above exception was the direct cause of the following exception: [e2e-predictor] [e2e-predictor] request = > [e2e-predictor] [e2e-predictor] @pytest.fixture(autouse=True) [e2e-predictor] def ensure_gateway_proxy_memory(request): [e2e-predictor] """After test setup creates gateways, patch them for proxy memory.""" [e2e-predictor] if not GATEWAY_PROXY_MEMORY: [e2e-predictor] return [e2e-predictor] [e2e-predictor] # Let test_case (llmisvc) create gateways first [e2e-predictor] [e2e-predictor] if "test_case" in request.fixturenames: [e2e-predictor] request.getfixturevalue("test_case") [e2e-predictor] [e2e-predictor] from kserve import KServeClient [e2e-predictor] [e2e-predictor] kserve_client = KServeClient( [e2e-predictor] config_file=os.environ.get("KUBECONFIG", "~/.kube/config") [e2e-predictor] ) [e2e-predictor] api_client = kserve_client.api_instance.api_client [e2e-predictor] [e2e-predictor] namespace = os.environ.get("KSERVE_TEST_NAMESPACE", "kserve-ci-e2e-test") [e2e-predictor] _ensure_configmap(namespace, api_client) [e2e-predictor] > patched = _patch_gateways(namespace, api_client) [e2e-predictor] [e2e-predictor] common/gateway_proxy_istio.py:194: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] namespace = 'kserve-ci-e2e-test' [e2e-predictor] api_client = [e2e-predictor] [e2e-predictor] def _patch_gateways(namespace, api_client): [e2e-predictor] """Patch all Gateways in namespace to include parametersRef. [e2e-predictor] [e2e-predictor] Returns the list of gateway names that were actually patched. [e2e-predictor] """ [e2e-predictor] custom_api = client.CustomObjectsApi(api_client) [e2e-predictor] > gateways = custom_api.list_namespaced_custom_object( [e2e-predictor] "gateway.networking.k8s.io", "v1", namespace, "gateways" [e2e-predictor] ) [e2e-predictor] [e2e-predictor] common/gateway_proxy_istio.py:117: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] group = 'gateway.networking.k8s.io', version = 'v1' [e2e-predictor] namespace = 'kserve-ci-e2e-test', plural = 'gateways' [e2e-predictor] kwargs = {'_return_http_data_only': True} [e2e-predictor] [e2e-predictor] def list_namespaced_custom_object(self, group, version, namespace, plural, **kwargs): # noqa: E501 [e2e-predictor] """list_namespaced_custom_object # noqa: E501 [e2e-predictor] [e2e-predictor] list or watch namespace scoped custom objects # noqa: E501 [e2e-predictor] This method makes a synchronous HTTP request by default. To make an [e2e-predictor] asynchronous HTTP request, please pass async_req=True [e2e-predictor] >>> thread = api.list_namespaced_custom_object(group, version, namespace, plural, async_req=True) [e2e-predictor] >>> result = thread.get() [e2e-predictor] [e2e-predictor] :param async_req bool: execute request asynchronously [e2e-predictor] :param str group: The custom resource's group name (required) [e2e-predictor] :param str version: The custom resource's version (required) [e2e-predictor] :param str namespace: The custom resource's namespace (required) [e2e-predictor] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-predictor] :param str pretty: If 'true', then the output is pretty printed. [e2e-predictor] :param bool allow_watch_bookmarks: allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored. [e2e-predictor] :param str _continue: The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\". This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications. [e2e-predictor] :param str field_selector: A selector to restrict the list of returned objects by their fields. Defaults to everything. [e2e-predictor] :param str label_selector: A selector to restrict the list of returned objects by their labels. Defaults to everything. [e2e-predictor] :param int limit: limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true. The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned. [e2e-predictor] :param str resource_version: When specified with a watch call, shows changes that occur after that particular version of a resource. Defaults to changes from the beginning of history. When specified for list: - if unset, then the result is returned from remote storage based on quorum-read flag; - if it's 0, then we simply return what we currently have in cache, no guarantee; - if set to non zero, then the result is at least as fresh as given rv. [e2e-predictor] :param str resource_version_match: resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details. Defaults to unset [e2e-predictor] :param int timeout_seconds: Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity. [e2e-predictor] :param bool watch: Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] :return: object [e2e-predictor] If the method is called asynchronously, [e2e-predictor] returns the request thread. [e2e-predictor] """ [e2e-predictor] kwargs['_return_http_data_only'] = True [e2e-predictor] > return self.list_namespaced_custom_object_with_http_info(group, version, namespace, plural, **kwargs) # noqa: E501 [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:2455: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] group = 'gateway.networking.k8s.io', version = 'v1' [e2e-predictor] namespace = 'kserve-ci-e2e-test', plural = 'gateways' [e2e-predictor] kwargs = {'_return_http_data_only': True} [e2e-predictor] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'pretty', 'allow_watch_bookmarks', ...], 'auth_settings': ['BearerToken'], 'body_params': None, ...} [e2e-predictor] all_params = ['group', 'version', 'namespace', 'plural', 'pretty', 'allow_watch_bookmarks', ...] [e2e-predictor] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-predictor] path_params = {'group': 'gateway.networking.k8s.io', 'namespace': 'kserve-ci-e2e-test', 'plural': 'gateways', 'version': 'v1'} [e2e-predictor] query_params = [] [e2e-predictor] [e2e-predictor] def list_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, **kwargs): # noqa: E501 [e2e-predictor] """list_namespaced_custom_object # noqa: E501 [e2e-predictor] [e2e-predictor] list or watch namespace scoped custom objects # noqa: E501 [e2e-predictor] This method makes a synchronous HTTP request by default. To make an [e2e-predictor] asynchronous HTTP request, please pass async_req=True [e2e-predictor] >>> thread = api.list_namespaced_custom_object_with_http_info(group, version, namespace, plural, async_req=True) [e2e-predictor] >>> result = thread.get() [e2e-predictor] [e2e-predictor] :param async_req bool: execute request asynchronously [e2e-predictor] :param str group: The custom resource's group name (required) [e2e-predictor] :param str version: The custom resource's version (required) [e2e-predictor] :param str namespace: The custom resource's namespace (required) [e2e-predictor] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-predictor] :param str pretty: If 'true', then the output is pretty printed. [e2e-predictor] :param bool allow_watch_bookmarks: allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored. [e2e-predictor] :param str _continue: The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\". This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications. [e2e-predictor] :param str field_selector: A selector to restrict the list of returned objects by their fields. Defaults to everything. [e2e-predictor] :param str label_selector: A selector to restrict the list of returned objects by their labels. Defaults to everything. [e2e-predictor] :param int limit: limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true. The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned. [e2e-predictor] :param str resource_version: When specified with a watch call, shows changes that occur after that particular version of a resource. Defaults to changes from the beginning of history. When specified for list: - if unset, then the result is returned from remote storage based on quorum-read flag; - if it's 0, then we simply return what we currently have in cache, no guarantee; - if set to non zero, then the result is at least as fresh as given rv. [e2e-predictor] :param str resource_version_match: resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details. Defaults to unset [e2e-predictor] :param int timeout_seconds: Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity. [e2e-predictor] :param bool watch: Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. [e2e-predictor] :param _return_http_data_only: response data without head status code [e2e-predictor] and headers [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-predictor] If the method is called asynchronously, [e2e-predictor] returns the request thread. [e2e-predictor] """ [e2e-predictor] [e2e-predictor] local_var_params = locals() [e2e-predictor] [e2e-predictor] all_params = [ [e2e-predictor] 'group', [e2e-predictor] 'version', [e2e-predictor] 'namespace', [e2e-predictor] 'plural', [e2e-predictor] 'pretty', [e2e-predictor] 'allow_watch_bookmarks', [e2e-predictor] '_continue', [e2e-predictor] 'field_selector', [e2e-predictor] 'label_selector', [e2e-predictor] 'limit', [e2e-predictor] 'resource_version', [e2e-predictor] 'resource_version_match', [e2e-predictor] 'timeout_seconds', [e2e-predictor] 'watch' [e2e-predictor] ] [e2e-predictor] all_params.extend( [e2e-predictor] [ [e2e-predictor] 'async_req', [e2e-predictor] '_return_http_data_only', [e2e-predictor] '_preload_content', [e2e-predictor] '_request_timeout' [e2e-predictor] ] [e2e-predictor] ) [e2e-predictor] [e2e-predictor] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-predictor] if key not in all_params: [e2e-predictor] raise ApiTypeError( [e2e-predictor] "Got an unexpected keyword argument '%s'" [e2e-predictor] " to method list_namespaced_custom_object" % key [e2e-predictor] ) [e2e-predictor] local_var_params[key] = val [e2e-predictor] del local_var_params['kwargs'] [e2e-predictor] # verify the required parameter 'group' is set [e2e-predictor] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['group'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `group` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] # verify the required parameter 'version' is set [e2e-predictor] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['version'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `version` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] # verify the required parameter 'namespace' is set [e2e-predictor] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['namespace'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `namespace` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] # verify the required parameter 'plural' is set [e2e-predictor] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['plural'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `plural` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] [e2e-predictor] collection_formats = {} [e2e-predictor] [e2e-predictor] path_params = {} [e2e-predictor] if 'group' in local_var_params: [e2e-predictor] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-predictor] if 'version' in local_var_params: [e2e-predictor] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-predictor] if 'namespace' in local_var_params: [e2e-predictor] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-predictor] if 'plural' in local_var_params: [e2e-predictor] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-predictor] [e2e-predictor] query_params = [] [e2e-predictor] if 'pretty' in local_var_params and local_var_params['pretty'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('pretty', local_var_params['pretty'])) # noqa: E501 [e2e-predictor] if 'allow_watch_bookmarks' in local_var_params and local_var_params['allow_watch_bookmarks'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('allowWatchBookmarks', local_var_params['allow_watch_bookmarks'])) # noqa: E501 [e2e-predictor] if '_continue' in local_var_params and local_var_params['_continue'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('continue', local_var_params['_continue'])) # noqa: E501 [e2e-predictor] if 'field_selector' in local_var_params and local_var_params['field_selector'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('fieldSelector', local_var_params['field_selector'])) # noqa: E501 [e2e-predictor] if 'label_selector' in local_var_params and local_var_params['label_selector'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('labelSelector', local_var_params['label_selector'])) # noqa: E501 [e2e-predictor] if 'limit' in local_var_params and local_var_params['limit'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('limit', local_var_params['limit'])) # noqa: E501 [e2e-predictor] if 'resource_version' in local_var_params and local_var_params['resource_version'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('resourceVersion', local_var_params['resource_version'])) # noqa: E501 [e2e-predictor] if 'resource_version_match' in local_var_params and local_var_params['resource_version_match'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('resourceVersionMatch', local_var_params['resource_version_match'])) # noqa: E501 [e2e-predictor] if 'timeout_seconds' in local_var_params and local_var_params['timeout_seconds'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('timeoutSeconds', local_var_params['timeout_seconds'])) # noqa: E501 [e2e-predictor] if 'watch' in local_var_params and local_var_params['watch'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('watch', local_var_params['watch'])) # noqa: E501 [e2e-predictor] [e2e-predictor] header_params = {} [e2e-predictor] [e2e-predictor] form_params = [] [e2e-predictor] local_var_files = {} [e2e-predictor] [e2e-predictor] body_params = None [e2e-predictor] # HTTP header `Accept` [e2e-predictor] header_params['Accept'] = self.api_client.select_header_accept( [e2e-predictor] ['application/json', 'application/json;stream=watch']) # noqa: E501 [e2e-predictor] [e2e-predictor] # Authentication setting [e2e-predictor] auth_settings = ['BearerToken'] # noqa: E501 [e2e-predictor] [e2e-predictor] > return self.api_client.call_api( [e2e-predictor] '/apis/{group}/{version}/namespaces/{namespace}/{plural}', 'GET', [e2e-predictor] path_params, [e2e-predictor] query_params, [e2e-predictor] header_params, [e2e-predictor] body=body_params, [e2e-predictor] post_params=form_params, [e2e-predictor] files=local_var_files, [e2e-predictor] response_type='object', # noqa: E501 [e2e-predictor] auth_settings=auth_settings, [e2e-predictor] async_req=local_var_params.get('async_req'), [e2e-predictor] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-predictor] _preload_content=local_var_params.get('_preload_content', True), [e2e-predictor] _request_timeout=local_var_params.get('_request_timeout'), [e2e-predictor] collection_formats=collection_formats) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:2594: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}' [e2e-predictor] method = 'GET' [e2e-predictor] path_params = {'group': 'gateway.networking.k8s.io', 'namespace': 'kserve-ci-e2e-test', 'plural': 'gateways', 'version': 'v1'} [e2e-predictor] query_params = [] [e2e-predictor] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] body = None, post_params = [], files = {}, response_type = 'object' [e2e-predictor] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-predictor] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-predictor] _host = None [e2e-predictor] [e2e-predictor] def call_api(self, resource_path, method, [e2e-predictor] path_params=None, query_params=None, header_params=None, [e2e-predictor] body=None, post_params=None, files=None, [e2e-predictor] response_type=None, auth_settings=None, async_req=None, [e2e-predictor] _return_http_data_only=None, collection_formats=None, [e2e-predictor] _preload_content=True, _request_timeout=None, _host=None): [e2e-predictor] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-predictor] [e2e-predictor] To make an async_req request, set the async_req parameter. [e2e-predictor] [e2e-predictor] :param resource_path: Path to method endpoint. [e2e-predictor] :param method: Method to call. [e2e-predictor] :param path_params: Path parameters in the url. [e2e-predictor] :param query_params: Query parameters in the url. [e2e-predictor] :param header_params: Header parameters to be [e2e-predictor] placed in the request header. [e2e-predictor] :param body: Request body. [e2e-predictor] :param post_params dict: Request post form parameters, [e2e-predictor] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-predictor] :param auth_settings list: Auth Settings names for the request. [e2e-predictor] :param response: Response data type. [e2e-predictor] :param files dict: key -> filename, value -> filepath, [e2e-predictor] for `multipart/form-data`. [e2e-predictor] :param async_req bool: execute request asynchronously [e2e-predictor] :param _return_http_data_only: response data without head status code [e2e-predictor] and headers [e2e-predictor] :param collection_formats: dict of collection formats for path, query, [e2e-predictor] header, and post parameters. [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] :return: [e2e-predictor] If async_req parameter is True, [e2e-predictor] the request will be called asynchronously. [e2e-predictor] The method will return the request thread. [e2e-predictor] If parameter async_req is False or missing, [e2e-predictor] then the method will return the response directly. [e2e-predictor] """ [e2e-predictor] if not async_req: [e2e-predictor] > return self.__call_api(resource_path, method, [e2e-predictor] path_params, query_params, header_params, [e2e-predictor] body, post_params, files, [e2e-predictor] response_type, auth_settings, [e2e-predictor] _return_http_data_only, collection_formats, [e2e-predictor] _preload_content, _request_timeout, _host) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] resource_path = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] method = 'GET' [e2e-predictor] path_params = [('group', 'gateway.networking.k8s.io'), ('version', 'v1'), ('namespace', 'kserve-ci-e2e-test'), ('plural', 'gateways')] [e2e-predictor] query_params = [] [e2e-predictor] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] body = None, post_params = [], files = {}, response_type = 'object' [e2e-predictor] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-predictor] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-predictor] _host = None [e2e-predictor] [e2e-predictor] def __call_api( [e2e-predictor] self, resource_path, method, path_params=None, [e2e-predictor] query_params=None, header_params=None, body=None, post_params=None, [e2e-predictor] files=None, response_type=None, auth_settings=None, [e2e-predictor] _return_http_data_only=None, collection_formats=None, [e2e-predictor] _preload_content=True, _request_timeout=None, _host=None): [e2e-predictor] [e2e-predictor] config = self.configuration [e2e-predictor] [e2e-predictor] # header parameters [e2e-predictor] header_params = header_params or {} [e2e-predictor] header_params.update(self.default_headers) [e2e-predictor] if self.cookie: [e2e-predictor] header_params['Cookie'] = self.cookie [e2e-predictor] if header_params: [e2e-predictor] header_params = self.sanitize_for_serialization(header_params) [e2e-predictor] header_params = dict(self.parameters_to_tuples(header_params, [e2e-predictor] collection_formats)) [e2e-predictor] [e2e-predictor] # path parameters [e2e-predictor] if path_params: [e2e-predictor] path_params = self.sanitize_for_serialization(path_params) [e2e-predictor] path_params = self.parameters_to_tuples(path_params, [e2e-predictor] collection_formats) [e2e-predictor] for k, v in path_params: [e2e-predictor] # specified safe chars, encode everything [e2e-predictor] resource_path = resource_path.replace( [e2e-predictor] '{%s}' % k, [e2e-predictor] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # query parameters [e2e-predictor] if query_params: [e2e-predictor] query_params = self.sanitize_for_serialization(query_params) [e2e-predictor] query_params = self.parameters_to_tuples(query_params, [e2e-predictor] collection_formats) [e2e-predictor] [e2e-predictor] # post parameters [e2e-predictor] if post_params or files: [e2e-predictor] post_params = post_params if post_params else [] [e2e-predictor] post_params = self.sanitize_for_serialization(post_params) [e2e-predictor] post_params = self.parameters_to_tuples(post_params, [e2e-predictor] collection_formats) [e2e-predictor] post_params.extend(self.files_parameters(files)) [e2e-predictor] [e2e-predictor] # auth setting [e2e-predictor] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-predictor] [e2e-predictor] # body [e2e-predictor] if body: [e2e-predictor] body = self.sanitize_for_serialization(body) [e2e-predictor] [e2e-predictor] # request url [e2e-predictor] if _host is None: [e2e-predictor] url = self.configuration.host + resource_path [e2e-predictor] else: [e2e-predictor] # use server/host defined in path or operation instead [e2e-predictor] url = _host + resource_path [e2e-predictor] [e2e-predictor] # perform request and return response [e2e-predictor] > response_data = self.request( [e2e-predictor] method, url, query_params=query_params, headers=header_params, [e2e-predictor] post_params=post_params, body=body, [e2e-predictor] _preload_content=_preload_content, [e2e-predictor] _request_timeout=_request_timeout) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] query_params = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] post_params = [], body = None, _preload_content = True, _request_timeout = None [e2e-predictor] [e2e-predictor] def request(self, method, url, query_params=None, headers=None, [e2e-predictor] post_params=None, body=None, _preload_content=True, [e2e-predictor] _request_timeout=None): [e2e-predictor] """Makes the HTTP request using RESTClient.""" [e2e-predictor] if method == "GET": [e2e-predictor] > return self.rest_client.GET(url, [e2e-predictor] query_params=query_params, [e2e-predictor] _preload_content=_preload_content, [e2e-predictor] _request_timeout=_request_timeout, [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:373: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] query_params = [], _preload_content = True, _request_timeout = None [e2e-predictor] [e2e-predictor] def GET(self, url, headers=None, query_params=None, _preload_content=True, [e2e-predictor] _request_timeout=None): [e2e-predictor] > return self.request("GET", url, [e2e-predictor] headers=headers, [e2e-predictor] _preload_content=_preload_content, [e2e-predictor] _request_timeout=_request_timeout, [e2e-predictor] query_params=query_params) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:244: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] query_params = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] body = None, post_params = {}, _preload_content = True, _request_timeout = None [e2e-predictor] [e2e-predictor] def request(self, method, url, query_params=None, headers=None, [e2e-predictor] body=None, post_params=None, _preload_content=True, [e2e-predictor] _request_timeout=None): [e2e-predictor] """Perform requests. [e2e-predictor] [e2e-predictor] :param method: http request method [e2e-predictor] :param url: http request url [e2e-predictor] :param query_params: query parameters in the url [e2e-predictor] :param headers: http request headers [e2e-predictor] :param body: request json body, for `application/json` [e2e-predictor] :param post_params: request post parameters, [e2e-predictor] `application/x-www-form-urlencoded` [e2e-predictor] and `multipart/form-data` [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] """ [e2e-predictor] method = method.upper() [e2e-predictor] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-predictor] 'PATCH', 'OPTIONS'] [e2e-predictor] [e2e-predictor] if post_params and body: [e2e-predictor] raise ApiValueError( [e2e-predictor] "body parameter cannot be used with post_params parameter." [e2e-predictor] ) [e2e-predictor] [e2e-predictor] post_params = post_params or {} [e2e-predictor] headers = headers or {} [e2e-predictor] [e2e-predictor] timeout = None [e2e-predictor] if _request_timeout: [e2e-predictor] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-predictor] timeout = urllib3.Timeout(total=_request_timeout) [e2e-predictor] elif (isinstance(_request_timeout, tuple) and [e2e-predictor] len(_request_timeout) == 2): [e2e-predictor] timeout = urllib3.Timeout( [e2e-predictor] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-predictor] [e2e-predictor] if 'Content-Type' not in headers: [e2e-predictor] headers['Content-Type'] = 'application/json' [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-predictor] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-predictor] if query_params: [e2e-predictor] url += '?' + urlencode(query_params) [e2e-predictor] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-predictor] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-predictor] if headers['Content-Type'] == 'application/json-patch+json': [e2e-predictor] if not isinstance(body, list): [e2e-predictor] headers['Content-Type'] = \ [e2e-predictor] 'application/strategic-merge-patch+json' [e2e-predictor] request_body = None [e2e-predictor] if body is not None: [e2e-predictor] request_body = json.dumps(body) [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] body=request_body, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] fields=post_params, [e2e-predictor] encode_multipart=False, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] elif headers['Content-Type'] == 'multipart/form-data': [e2e-predictor] # must del headers['Content-Type'], or the correct [e2e-predictor] # Content-Type which generated by urllib3 will be [e2e-predictor] # overwritten. [e2e-predictor] del headers['Content-Type'] [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] fields=post_params, [e2e-predictor] encode_multipart=True, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] # Pass a `string` parameter directly in the body to support [e2e-predictor] # other content types than Json when `body` argument is [e2e-predictor] # provided in serialized form [e2e-predictor] elif isinstance(body, str) or isinstance(body, bytes): [e2e-predictor] request_body = body [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] body=request_body, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] else: [e2e-predictor] # Cannot generate the request from given parameters [e2e-predictor] msg = """Cannot prepare a request message for provided [e2e-predictor] arguments. Please check that your arguments match [e2e-predictor] declared content type.""" [e2e-predictor] raise ApiException(status=0, reason=msg) [e2e-predictor] # For `GET`, `HEAD` [e2e-predictor] else: [e2e-predictor] > r = self.pool_manager.request(method, url, [e2e-predictor] fields=query_params, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:217: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None, fields = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] json = None, urlopen_kw = {'preload_content': True, 'timeout': None} [e2e-predictor] [e2e-predictor] def request( [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] fields: _TYPE_FIELDS | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] json: typing.Any | None = None, [e2e-predictor] **urlopen_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Make a request using :meth:`urlopen` with the appropriate encoding of [e2e-predictor] ``fields`` based on the ``method`` used. [e2e-predictor] [e2e-predictor] This is a convenience method that requires the least amount of manual [e2e-predictor] effort. It can be used in most situations, while still having the [e2e-predictor] option to drop down to more specific methods when necessary, such as [e2e-predictor] :meth:`request_encode_url`, :meth:`request_encode_body`, [e2e-predictor] or even the lowest level :meth:`urlopen`. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param fields: [e2e-predictor] Data to encode and send in the URL or request body, depending on ``method``. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param json: [e2e-predictor] Data to encode and send as JSON with UTF-encoded in the request body. [e2e-predictor] The ``"Content-Type"`` header will be set to ``"application/json"`` [e2e-predictor] unless specified otherwise. [e2e-predictor] """ [e2e-predictor] method = method.upper() [e2e-predictor] [e2e-predictor] if json is not None and body is not None: [e2e-predictor] raise TypeError( [e2e-predictor] "request got values for both 'body' and 'json' parameters which are mutually exclusive" [e2e-predictor] ) [e2e-predictor] [e2e-predictor] if json is not None: [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not ("content-type" in map(str.lower, headers.keys())): [e2e-predictor] headers = HTTPHeaderDict(headers) [e2e-predictor] headers["Content-Type"] = "application/json" [e2e-predictor] [e2e-predictor] body = _json.dumps(json, separators=(",", ":"), ensure_ascii=False).encode( [e2e-predictor] "utf-8" [e2e-predictor] ) [e2e-predictor] [e2e-predictor] if body is not None: [e2e-predictor] urlopen_kw["body"] = body [e2e-predictor] [e2e-predictor] if method in self._encode_url_methods: [e2e-predictor] > return self.request_encode_url( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] fields=fields, # type: ignore[arg-type] [e2e-predictor] headers=headers, [e2e-predictor] **urlopen_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/_request_methods.py:135: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] fields = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] urlopen_kw = {'preload_content': True, 'timeout': None} [e2e-predictor] extra_kw = {'headers': {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'}, 'preload_content': True, 'timeout': None} [e2e-predictor] [e2e-predictor] def request_encode_url( [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] fields: _TYPE_ENCODE_URL_FIELDS | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] **urlopen_kw: str, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Make a request using :meth:`urlopen` with the ``fields`` encoded in [e2e-predictor] the url. This is useful for request methods like GET, HEAD, DELETE, etc. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param fields: [e2e-predictor] Data to encode and send in the URL. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] """ [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] extra_kw: dict[str, typing.Any] = {"headers": headers} [e2e-predictor] extra_kw.update(urlopen_kw) [e2e-predictor] [e2e-predictor] if fields: [e2e-predictor] url += "?" + urlencode(fields) [e2e-predictor] [e2e-predictor] > return self.urlopen(method, url, **extra_kw) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/_request_methods.py:182: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] redirect = True [e2e-predictor] kw = {'assert_same_host': False, 'headers': {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'}, 'preload_content': True, 'redirect': False, ...} [e2e-predictor] u = Url(scheme='https', auth=None, host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443, path='/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways', query=None, fragment=None) [e2e-predictor] conn = [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, method: str, url: str, redirect: bool = True, **kw: typing.Any [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Same as :meth:`urllib3.HTTPConnectionPool.urlopen` [e2e-predictor] with custom cross-host redirect logic and only sends the request-uri [e2e-predictor] portion of the ``url``. [e2e-predictor] [e2e-predictor] The given ``url`` parameter must be absolute, such that an appropriate [e2e-predictor] :class:`urllib3.connectionpool.ConnectionPool` can be chosen for it. [e2e-predictor] """ [e2e-predictor] u = parse_url(url) [e2e-predictor] [e2e-predictor] if u.scheme is None: [e2e-predictor] warnings.warn( [e2e-predictor] "URLs without a scheme (ie 'https://') are deprecated and will raise an error " [e2e-predictor] "in urllib3 v3.0. To avoid this FutureWarning ensure all URLs " [e2e-predictor] "start with 'https://' or 'http://'. Read more in this issue: " [e2e-predictor] "https://github.com/urllib3/urllib3/issues/2920", [e2e-predictor] category=FutureWarning, [e2e-predictor] stacklevel=2, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme) [e2e-predictor] [e2e-predictor] kw["assert_same_host"] = False [e2e-predictor] kw["redirect"] = False [e2e-predictor] [e2e-predictor] if "headers" not in kw: [e2e-predictor] kw["headers"] = self.headers [e2e-predictor] [e2e-predictor] if self._proxy_requires_url_absolute_form(u): [e2e-predictor] response = conn.urlopen(method, url, **kw) [e2e-predictor] else: [e2e-predictor] > response = conn.urlopen(method, u.request_uri, **kw) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/poolmanager.py:457: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=2, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False [e2e-predictor] err = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] retries.sleep() [e2e-predictor] [e2e-predictor] # Keep track of the error for the retry warning. [e2e-predictor] err = e [e2e-predictor] [e2e-predictor] finally: [e2e-predictor] if not clean_exit: [e2e-predictor] # We hit some kind of exception, handled or otherwise. We need [e2e-predictor] # to throw the connection away unless explicitly told not to. [e2e-predictor] # Close the connection, set the variable to None, and make sure [e2e-predictor] # we put the None back in the pool to avoid leaking it. [e2e-predictor] if conn: [e2e-predictor] conn.close() [e2e-predictor] conn = None [e2e-predictor] release_this_conn = True [e2e-predictor] [e2e-predictor] if release_this_conn: [e2e-predictor] # Put the connection back to be reused. If the connection is [e2e-predictor] # expired then it will be None, which will get replaced with a [e2e-predictor] # fresh connection during _get_conn. [e2e-predictor] self._put_conn(conn) [e2e-predictor] [e2e-predictor] if not conn: [e2e-predictor] # Try again [e2e-predictor] log.warning( [e2e-predictor] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-predictor] ) [e2e-predictor] > return self.urlopen( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] body, [e2e-predictor] headers, [e2e-predictor] retries, [e2e-predictor] redirect, [e2e-predictor] assert_same_host, [e2e-predictor] timeout=timeout, [e2e-predictor] pool_timeout=pool_timeout, [e2e-predictor] release_conn=release_conn, [e2e-predictor] chunked=chunked, [e2e-predictor] body_pos=body_pos, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:872: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=1, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False [e2e-predictor] err = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] retries.sleep() [e2e-predictor] [e2e-predictor] # Keep track of the error for the retry warning. [e2e-predictor] err = e [e2e-predictor] [e2e-predictor] finally: [e2e-predictor] if not clean_exit: [e2e-predictor] # We hit some kind of exception, handled or otherwise. We need [e2e-predictor] # to throw the connection away unless explicitly told not to. [e2e-predictor] # Close the connection, set the variable to None, and make sure [e2e-predictor] # we put the None back in the pool to avoid leaking it. [e2e-predictor] if conn: [e2e-predictor] conn.close() [e2e-predictor] conn = None [e2e-predictor] release_this_conn = True [e2e-predictor] [e2e-predictor] if release_this_conn: [e2e-predictor] # Put the connection back to be reused. If the connection is [e2e-predictor] # expired then it will be None, which will get replaced with a [e2e-predictor] # fresh connection during _get_conn. [e2e-predictor] self._put_conn(conn) [e2e-predictor] [e2e-predictor] if not conn: [e2e-predictor] # Try again [e2e-predictor] log.warning( [e2e-predictor] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-predictor] ) [e2e-predictor] > return self.urlopen( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] body, [e2e-predictor] headers, [e2e-predictor] retries, [e2e-predictor] redirect, [e2e-predictor] assert_same_host, [e2e-predictor] timeout=timeout, [e2e-predictor] pool_timeout=pool_timeout, [e2e-predictor] release_conn=release_conn, [e2e-predictor] chunked=chunked, [e2e-predictor] body_pos=body_pos, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:872: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False [e2e-predictor] err = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] retries.sleep() [e2e-predictor] [e2e-predictor] # Keep track of the error for the retry warning. [e2e-predictor] err = e [e2e-predictor] [e2e-predictor] finally: [e2e-predictor] if not clean_exit: [e2e-predictor] # We hit some kind of exception, handled or otherwise. We need [e2e-predictor] # to throw the connection away unless explicitly told not to. [e2e-predictor] # Close the connection, set the variable to None, and make sure [e2e-predictor] # we put the None back in the pool to avoid leaking it. [e2e-predictor] if conn: [e2e-predictor] conn.close() [e2e-predictor] conn = None [e2e-predictor] release_this_conn = True [e2e-predictor] [e2e-predictor] if release_this_conn: [e2e-predictor] # Put the connection back to be reused. If the connection is [e2e-predictor] # expired then it will be None, which will get replaced with a [e2e-predictor] # fresh connection during _get_conn. [e2e-predictor] self._put_conn(conn) [e2e-predictor] [e2e-predictor] if not conn: [e2e-predictor] # Try again [e2e-predictor] log.warning( [e2e-predictor] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-predictor] ) [e2e-predictor] > return self.urlopen( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] body, [e2e-predictor] headers, [e2e-predictor] retries, [e2e-predictor] redirect, [e2e-predictor] assert_same_host, [e2e-predictor] timeout=timeout, [e2e-predictor] pool_timeout=pool_timeout, [e2e-predictor] release_conn=release_conn, [e2e-predictor] chunked=chunked, [e2e-predictor] body_pos=body_pos, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:872: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False, err = None [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] > retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:842: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] response = None [e2e-predictor] error = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] _pool = [e2e-predictor] _stacktrace = [e2e-predictor] [e2e-predictor] def increment( [e2e-predictor] self, [e2e-predictor] method: str | None = None, [e2e-predictor] url: str | None = None, [e2e-predictor] response: BaseHTTPResponse | None = None, [e2e-predictor] error: Exception | None = None, [e2e-predictor] _pool: ConnectionPool | None = None, [e2e-predictor] _stacktrace: TracebackType | None = None, [e2e-predictor] ) -> Self: [e2e-predictor] """Return a new Retry object with incremented retry counters. [e2e-predictor] [e2e-predictor] :param response: A response object, or None, if the server did not [e2e-predictor] return a response. [e2e-predictor] :type response: :class:`~urllib3.response.BaseHTTPResponse` [e2e-predictor] :param Exception error: An error encountered during the request, or [e2e-predictor] None if the response was received successfully. [e2e-predictor] [e2e-predictor] :return: A new ``Retry`` object. [e2e-predictor] """ [e2e-predictor] if self.total is False and error: [e2e-predictor] # Disabled, indicate to re-raise the error. [e2e-predictor] raise reraise(type(error), error, _stacktrace) [e2e-predictor] [e2e-predictor] total = self.total [e2e-predictor] if total is not None: [e2e-predictor] total -= 1 [e2e-predictor] [e2e-predictor] connect = self.connect [e2e-predictor] read = self.read [e2e-predictor] redirect = self.redirect [e2e-predictor] status_count = self.status [e2e-predictor] other = self.other [e2e-predictor] cause = "unknown" [e2e-predictor] status = None [e2e-predictor] redirect_location = None [e2e-predictor] [e2e-predictor] if error and self._is_connection_error(error): [e2e-predictor] # Connect retry? [e2e-predictor] if connect is False: [e2e-predictor] raise reraise(type(error), error, _stacktrace) [e2e-predictor] elif connect is not None: [e2e-predictor] connect -= 1 [e2e-predictor] [e2e-predictor] elif error and self._is_read_error(error): [e2e-predictor] # Read retry? [e2e-predictor] if read is False or method is None or not self._is_method_retryable(method): [e2e-predictor] raise reraise(type(error), error, _stacktrace) [e2e-predictor] elif read is not None: [e2e-predictor] read -= 1 [e2e-predictor] [e2e-predictor] elif error: [e2e-predictor] # Other retry? [e2e-predictor] if other is not None: [e2e-predictor] other -= 1 [e2e-predictor] [e2e-predictor] elif response and response.get_redirect_location(): [e2e-predictor] # Redirect retry? [e2e-predictor] if redirect is not None: [e2e-predictor] redirect -= 1 [e2e-predictor] cause = "too many redirects" [e2e-predictor] response_redirect_location = response.get_redirect_location() [e2e-predictor] if response_redirect_location: [e2e-predictor] redirect_location = response_redirect_location [e2e-predictor] status = response.status [e2e-predictor] [e2e-predictor] else: [e2e-predictor] # Incrementing because of a server error like a 500 in [e2e-predictor] # status_forcelist and the given method is in the allowed_methods [e2e-predictor] cause = ResponseError.GENERIC_ERROR [e2e-predictor] if response and response.status: [e2e-predictor] if status_count is not None: [e2e-predictor] status_count -= 1 [e2e-predictor] cause = ResponseError.SPECIFIC_ERROR.format(status_code=response.status) [e2e-predictor] status = response.status [e2e-predictor] [e2e-predictor] history = self.history + ( [e2e-predictor] RequestHistory(method, url, error, status, redirect_location), [e2e-predictor] ) [e2e-predictor] [e2e-predictor] new_retry = self.new( [e2e-predictor] total=total, [e2e-predictor] connect=connect, [e2e-predictor] read=read, [e2e-predictor] redirect=redirect, [e2e-predictor] status=status_count, [e2e-predictor] other=other, [e2e-predictor] history=history, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] if new_retry.is_exhausted(): [e2e-predictor] reason = error or ResponseError(cause) [e2e-predictor] > raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] [e2e-predictor] E urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Max retries exceeded with url: /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways (Caused by NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/retry.py:543: MaxRetryError [e2e-predictor] ------------------------------ Captured log setup ------------------------------ [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways [e2e-predictor] ______________________ ERROR at setup of test_sklearn_v2 _______________________ [e2e-predictor] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-predictor] [e2e-predictor] self = [e2e-predictor] [e2e-predictor] def _new_conn(self) -> socket.socket: [e2e-predictor] """Establish a socket connection and set nodelay settings on it. [e2e-predictor] [e2e-predictor] :return: New socket connection. [e2e-predictor] """ [e2e-predictor] try: [e2e-predictor] > sock = connection.create_connection( [e2e-predictor] (self._dns_host, self.port), [e2e-predictor] self.timeout, [e2e-predictor] source_address=self.source_address, [e2e-predictor] socket_options=self.socket_options, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:204: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] address = ('a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', 6443) [e2e-predictor] timeout = None, source_address = None, socket_options = [(6, 1, 1)] [e2e-predictor] [e2e-predictor] def create_connection( [e2e-predictor] address: tuple[str, int], [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] source_address: tuple[str, int] | None = None, [e2e-predictor] socket_options: _TYPE_SOCKET_OPTIONS | None = None, [e2e-predictor] ) -> socket.socket: [e2e-predictor] """Connect to *address* and return the socket object. [e2e-predictor] [e2e-predictor] Convenience function. Connect to *address* (a 2-tuple ``(host, [e2e-predictor] port)``) and return the socket object. Passing the optional [e2e-predictor] *timeout* parameter will set the timeout on the socket instance [e2e-predictor] before attempting to connect. If no *timeout* is supplied, the [e2e-predictor] global default timeout setting returned by :func:`socket.getdefaulttimeout` [e2e-predictor] is used. If *source_address* is set it must be a tuple of (host, port) [e2e-predictor] for the socket to bind as a source address before making the connection. [e2e-predictor] An host of '' or port 0 tells the OS to use the default. [e2e-predictor] """ [e2e-predictor] [e2e-predictor] host, port = address [e2e-predictor] if host.startswith("["): [e2e-predictor] host = host.strip("[]") [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Using the value from allowed_gai_family() in the context of getaddrinfo lets [e2e-predictor] # us select whether to work with IPv4 DNS records, IPv6 records, or both. [e2e-predictor] # The original create_connection function always returns all records. [e2e-predictor] family = allowed_gai_family() [e2e-predictor] [e2e-predictor] try: [e2e-predictor] host.encode("idna") [e2e-predictor] except UnicodeError: [e2e-predictor] raise LocationParseError(f"'{host}', label empty or too long") from None [e2e-predictor] [e2e-predictor] > for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/connection.py:60: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] host = 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' [e2e-predictor] port = 6443, family = [e2e-predictor] type = , proto = 0, flags = 0 [e2e-predictor] [e2e-predictor] def getaddrinfo(host, port, family=0, type=0, proto=0, flags=0): [e2e-predictor] """Resolve host and port into list of address info entries. [e2e-predictor] [e2e-predictor] Translate the host/port argument into a sequence of 5-tuples that contain [e2e-predictor] all the necessary arguments for creating a socket connected to that service. [e2e-predictor] host is a domain name, a string representation of an IPv4/v6 address or [e2e-predictor] None. port is a string service name such as 'http', a numeric port number or [e2e-predictor] None. By passing None as the value of host and port, you can pass NULL to [e2e-predictor] the underlying C API. [e2e-predictor] [e2e-predictor] The family, type and proto arguments can be optionally specified in order to [e2e-predictor] narrow the list of addresses returned. Passing zero as a value for each of [e2e-predictor] these arguments selects the full range of results. [e2e-predictor] """ [e2e-predictor] # We override this function since we want to translate the numeric family [e2e-predictor] # and socket type values to enum constants. [e2e-predictor] addrlist = [] [e2e-predictor] > for res in _socket.getaddrinfo(host, port, family, type, proto, flags): [e2e-predictor] E socket.gaierror: [Errno -2] Name or service not known [e2e-predictor] [e2e-predictor] /usr/lib64/python3.11/socket.py:974: gaierror [e2e-predictor] [e2e-predictor] The above exception was the direct cause of the following exception: [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False, err = None [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] > response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:788: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] conn = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] timeout = Timeout(connect=None, read=None, total=None), chunked = False [e2e-predictor] response_conn = None, preload_content = True, decode_content = True [e2e-predictor] enforce_content_length = True [e2e-predictor] [e2e-predictor] def _make_request( [e2e-predictor] self, [e2e-predictor] conn: BaseHTTPConnection, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | None = None, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] chunked: bool = False, [e2e-predictor] response_conn: BaseHTTPConnection | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] enforce_content_length: bool = True, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Perform a request on a given urllib connection object taken from our [e2e-predictor] pool. [e2e-predictor] [e2e-predictor] :param conn: [e2e-predictor] a connection from one of our connection pools [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] Pass ``None`` to retry until you receive a response. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param response_conn: [e2e-predictor] Set this to ``None`` if you will handle releasing the connection or [e2e-predictor] set the connection to have the response release it. [e2e-predictor] [e2e-predictor] :param preload_content: [e2e-predictor] If True, the response's body will be preloaded during construction. [e2e-predictor] [e2e-predictor] :param decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param enforce_content_length: [e2e-predictor] Enforce content length checking. Body returned by server must match [e2e-predictor] value of Content-Length header, if present. Otherwise, raise error. [e2e-predictor] """ [e2e-predictor] self.num_requests += 1 [e2e-predictor] [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] timeout_obj.start_connect() [e2e-predictor] conn.timeout = Timeout.resolve_default_timeout(timeout_obj.connect_timeout) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Trigger any extra validation we need to do. [e2e-predictor] try: [e2e-predictor] self._validate_conn(conn) [e2e-predictor] except (SocketTimeout, BaseSSLError) as e: [e2e-predictor] self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # _validate_conn() starts the connection to an HTTPS proxy [e2e-predictor] # so we need to wrap errors with 'ProxyError' here too. [e2e-predictor] except ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] BaseSSLError, [e2e-predictor] CertificateError, [e2e-predictor] SSLError, [e2e-predictor] ) as e: [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] # If the connection didn't successfully connect to it's proxy [e2e-predictor] # then there [e2e-predictor] if isinstance( [e2e-predictor] new_e, (OSError, NewConnectionError, TimeoutError, SSLError) [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] > raise new_e [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:488: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] conn = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] timeout = Timeout(connect=None, read=None, total=None), chunked = False [e2e-predictor] response_conn = None, preload_content = True, decode_content = True [e2e-predictor] enforce_content_length = True [e2e-predictor] [e2e-predictor] def _make_request( [e2e-predictor] self, [e2e-predictor] conn: BaseHTTPConnection, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | None = None, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] chunked: bool = False, [e2e-predictor] response_conn: BaseHTTPConnection | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] enforce_content_length: bool = True, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Perform a request on a given urllib connection object taken from our [e2e-predictor] pool. [e2e-predictor] [e2e-predictor] :param conn: [e2e-predictor] a connection from one of our connection pools [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] Pass ``None`` to retry until you receive a response. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param response_conn: [e2e-predictor] Set this to ``None`` if you will handle releasing the connection or [e2e-predictor] set the connection to have the response release it. [e2e-predictor] [e2e-predictor] :param preload_content: [e2e-predictor] If True, the response's body will be preloaded during construction. [e2e-predictor] [e2e-predictor] :param decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param enforce_content_length: [e2e-predictor] Enforce content length checking. Body returned by server must match [e2e-predictor] value of Content-Length header, if present. Otherwise, raise error. [e2e-predictor] """ [e2e-predictor] self.num_requests += 1 [e2e-predictor] [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] timeout_obj.start_connect() [e2e-predictor] conn.timeout = Timeout.resolve_default_timeout(timeout_obj.connect_timeout) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Trigger any extra validation we need to do. [e2e-predictor] try: [e2e-predictor] > self._validate_conn(conn) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:464: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] conn = [e2e-predictor] [e2e-predictor] def _validate_conn(self, conn: BaseHTTPConnection) -> None: [e2e-predictor] """ [e2e-predictor] Called right before a request is made, after the socket is created. [e2e-predictor] """ [e2e-predictor] super()._validate_conn(conn) [e2e-predictor] [e2e-predictor] # Force connect early to allow us to validate the connection. [e2e-predictor] if conn.is_closed: [e2e-predictor] > conn.connect() [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:1106: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] [e2e-predictor] def connect(self) -> None: [e2e-predictor] # Today we don't need to be doing this step before the /actual/ socket [e2e-predictor] # connection, however in the future we'll need to decide whether to [e2e-predictor] # create a new socket or re-use an existing "shared" socket as a part [e2e-predictor] # of the HTTP/2 handshake dance. [e2e-predictor] if self._tunnel_host is not None and self._tunnel_port is not None: [e2e-predictor] probe_http2_host = self._tunnel_host [e2e-predictor] probe_http2_port = self._tunnel_port [e2e-predictor] else: [e2e-predictor] probe_http2_host = self.host [e2e-predictor] probe_http2_port = self.port [e2e-predictor] [e2e-predictor] # Check if the target origin supports HTTP/2. [e2e-predictor] # If the value comes back as 'None' it means that the current thread [e2e-predictor] # is probing for HTTP/2 support. Otherwise, we're waiting for another [e2e-predictor] # probe to complete, or we get a value right away. [e2e-predictor] target_supports_http2: bool | None [e2e-predictor] if "h2" in ssl_.ALPN_PROTOCOLS: [e2e-predictor] target_supports_http2 = http2_probe.acquire_and_get( [e2e-predictor] host=probe_http2_host, port=probe_http2_port [e2e-predictor] ) [e2e-predictor] else: [e2e-predictor] # If HTTP/2 isn't going to be offered it doesn't matter if [e2e-predictor] # the target supports HTTP/2. Don't want to make a probe. [e2e-predictor] target_supports_http2 = False [e2e-predictor] [e2e-predictor] if self._connect_callback is not None: [e2e-predictor] self._connect_callback( [e2e-predictor] "before connect", [e2e-predictor] thread_id=threading.get_ident(), [e2e-predictor] target_supports_http2=target_supports_http2, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] sock: socket.socket | ssl.SSLSocket [e2e-predictor] > self.sock = sock = self._new_conn() [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:759: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] [e2e-predictor] def _new_conn(self) -> socket.socket: [e2e-predictor] """Establish a socket connection and set nodelay settings on it. [e2e-predictor] [e2e-predictor] :return: New socket connection. [e2e-predictor] """ [e2e-predictor] try: [e2e-predictor] sock = connection.create_connection( [e2e-predictor] (self._dns_host, self.port), [e2e-predictor] self.timeout, [e2e-predictor] source_address=self.source_address, [e2e-predictor] socket_options=self.socket_options, [e2e-predictor] ) [e2e-predictor] except socket.gaierror as e: [e2e-predictor] > raise NameResolutionError(self.host, self, e) from e [e2e-predictor] E urllib3.exceptions.NameResolutionError: HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:211: NameResolutionError [e2e-predictor] [e2e-predictor] The above exception was the direct cause of the following exception: [e2e-predictor] [e2e-predictor] request = > [e2e-predictor] [e2e-predictor] @pytest.fixture(autouse=True) [e2e-predictor] def ensure_gateway_proxy_memory(request): [e2e-predictor] """After test setup creates gateways, patch them for proxy memory.""" [e2e-predictor] if not GATEWAY_PROXY_MEMORY: [e2e-predictor] return [e2e-predictor] [e2e-predictor] # Let test_case (llmisvc) create gateways first [e2e-predictor] [e2e-predictor] if "test_case" in request.fixturenames: [e2e-predictor] request.getfixturevalue("test_case") [e2e-predictor] [e2e-predictor] from kserve import KServeClient [e2e-predictor] [e2e-predictor] kserve_client = KServeClient( [e2e-predictor] config_file=os.environ.get("KUBECONFIG", "~/.kube/config") [e2e-predictor] ) [e2e-predictor] api_client = kserve_client.api_instance.api_client [e2e-predictor] [e2e-predictor] namespace = os.environ.get("KSERVE_TEST_NAMESPACE", "kserve-ci-e2e-test") [e2e-predictor] _ensure_configmap(namespace, api_client) [e2e-predictor] > patched = _patch_gateways(namespace, api_client) [e2e-predictor] [e2e-predictor] common/gateway_proxy_istio.py:194: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] namespace = 'kserve-ci-e2e-test' [e2e-predictor] api_client = [e2e-predictor] [e2e-predictor] def _patch_gateways(namespace, api_client): [e2e-predictor] """Patch all Gateways in namespace to include parametersRef. [e2e-predictor] [e2e-predictor] Returns the list of gateway names that were actually patched. [e2e-predictor] """ [e2e-predictor] custom_api = client.CustomObjectsApi(api_client) [e2e-predictor] > gateways = custom_api.list_namespaced_custom_object( [e2e-predictor] "gateway.networking.k8s.io", "v1", namespace, "gateways" [e2e-predictor] ) [e2e-predictor] [e2e-predictor] common/gateway_proxy_istio.py:117: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] group = 'gateway.networking.k8s.io', version = 'v1' [e2e-predictor] namespace = 'kserve-ci-e2e-test', plural = 'gateways' [e2e-predictor] kwargs = {'_return_http_data_only': True} [e2e-predictor] [e2e-predictor] def list_namespaced_custom_object(self, group, version, namespace, plural, **kwargs): # noqa: E501 [e2e-predictor] """list_namespaced_custom_object # noqa: E501 [e2e-predictor] [e2e-predictor] list or watch namespace scoped custom objects # noqa: E501 [e2e-predictor] This method makes a synchronous HTTP request by default. To make an [e2e-predictor] asynchronous HTTP request, please pass async_req=True [e2e-predictor] >>> thread = api.list_namespaced_custom_object(group, version, namespace, plural, async_req=True) [e2e-predictor] >>> result = thread.get() [e2e-predictor] [e2e-predictor] :param async_req bool: execute request asynchronously [e2e-predictor] :param str group: The custom resource's group name (required) [e2e-predictor] :param str version: The custom resource's version (required) [e2e-predictor] :param str namespace: The custom resource's namespace (required) [e2e-predictor] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-predictor] :param str pretty: If 'true', then the output is pretty printed. [e2e-predictor] :param bool allow_watch_bookmarks: allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored. [e2e-predictor] :param str _continue: The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\". This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications. [e2e-predictor] :param str field_selector: A selector to restrict the list of returned objects by their fields. Defaults to everything. [e2e-predictor] :param str label_selector: A selector to restrict the list of returned objects by their labels. Defaults to everything. [e2e-predictor] :param int limit: limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true. The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned. [e2e-predictor] :param str resource_version: When specified with a watch call, shows changes that occur after that particular version of a resource. Defaults to changes from the beginning of history. When specified for list: - if unset, then the result is returned from remote storage based on quorum-read flag; - if it's 0, then we simply return what we currently have in cache, no guarantee; - if set to non zero, then the result is at least as fresh as given rv. [e2e-predictor] :param str resource_version_match: resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details. Defaults to unset [e2e-predictor] :param int timeout_seconds: Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity. [e2e-predictor] :param bool watch: Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] :return: object [e2e-predictor] If the method is called asynchronously, [e2e-predictor] returns the request thread. [e2e-predictor] """ [e2e-predictor] kwargs['_return_http_data_only'] = True [e2e-predictor] > return self.list_namespaced_custom_object_with_http_info(group, version, namespace, plural, **kwargs) # noqa: E501 [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:2455: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] group = 'gateway.networking.k8s.io', version = 'v1' [e2e-predictor] namespace = 'kserve-ci-e2e-test', plural = 'gateways' [e2e-predictor] kwargs = {'_return_http_data_only': True} [e2e-predictor] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'pretty', 'allow_watch_bookmarks', ...], 'auth_settings': ['BearerToken'], 'body_params': None, ...} [e2e-predictor] all_params = ['group', 'version', 'namespace', 'plural', 'pretty', 'allow_watch_bookmarks', ...] [e2e-predictor] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-predictor] path_params = {'group': 'gateway.networking.k8s.io', 'namespace': 'kserve-ci-e2e-test', 'plural': 'gateways', 'version': 'v1'} [e2e-predictor] query_params = [] [e2e-predictor] [e2e-predictor] def list_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, **kwargs): # noqa: E501 [e2e-predictor] """list_namespaced_custom_object # noqa: E501 [e2e-predictor] [e2e-predictor] list or watch namespace scoped custom objects # noqa: E501 [e2e-predictor] This method makes a synchronous HTTP request by default. To make an [e2e-predictor] asynchronous HTTP request, please pass async_req=True [e2e-predictor] >>> thread = api.list_namespaced_custom_object_with_http_info(group, version, namespace, plural, async_req=True) [e2e-predictor] >>> result = thread.get() [e2e-predictor] [e2e-predictor] :param async_req bool: execute request asynchronously [e2e-predictor] :param str group: The custom resource's group name (required) [e2e-predictor] :param str version: The custom resource's version (required) [e2e-predictor] :param str namespace: The custom resource's namespace (required) [e2e-predictor] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-predictor] :param str pretty: If 'true', then the output is pretty printed. [e2e-predictor] :param bool allow_watch_bookmarks: allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored. [e2e-predictor] :param str _continue: The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\". This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications. [e2e-predictor] :param str field_selector: A selector to restrict the list of returned objects by their fields. Defaults to everything. [e2e-predictor] :param str label_selector: A selector to restrict the list of returned objects by their labels. Defaults to everything. [e2e-predictor] :param int limit: limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true. The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned. [e2e-predictor] :param str resource_version: When specified with a watch call, shows changes that occur after that particular version of a resource. Defaults to changes from the beginning of history. When specified for list: - if unset, then the result is returned from remote storage based on quorum-read flag; - if it's 0, then we simply return what we currently have in cache, no guarantee; - if set to non zero, then the result is at least as fresh as given rv. [e2e-predictor] :param str resource_version_match: resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details. Defaults to unset [e2e-predictor] :param int timeout_seconds: Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity. [e2e-predictor] :param bool watch: Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. [e2e-predictor] :param _return_http_data_only: response data without head status code [e2e-predictor] and headers [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-predictor] If the method is called asynchronously, [e2e-predictor] returns the request thread. [e2e-predictor] """ [e2e-predictor] [e2e-predictor] local_var_params = locals() [e2e-predictor] [e2e-predictor] all_params = [ [e2e-predictor] 'group', [e2e-predictor] 'version', [e2e-predictor] 'namespace', [e2e-predictor] 'plural', [e2e-predictor] 'pretty', [e2e-predictor] 'allow_watch_bookmarks', [e2e-predictor] '_continue', [e2e-predictor] 'field_selector', [e2e-predictor] 'label_selector', [e2e-predictor] 'limit', [e2e-predictor] 'resource_version', [e2e-predictor] 'resource_version_match', [e2e-predictor] 'timeout_seconds', [e2e-predictor] 'watch' [e2e-predictor] ] [e2e-predictor] all_params.extend( [e2e-predictor] [ [e2e-predictor] 'async_req', [e2e-predictor] '_return_http_data_only', [e2e-predictor] '_preload_content', [e2e-predictor] '_request_timeout' [e2e-predictor] ] [e2e-predictor] ) [e2e-predictor] [e2e-predictor] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-predictor] if key not in all_params: [e2e-predictor] raise ApiTypeError( [e2e-predictor] "Got an unexpected keyword argument '%s'" [e2e-predictor] " to method list_namespaced_custom_object" % key [e2e-predictor] ) [e2e-predictor] local_var_params[key] = val [e2e-predictor] del local_var_params['kwargs'] [e2e-predictor] # verify the required parameter 'group' is set [e2e-predictor] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['group'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `group` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] # verify the required parameter 'version' is set [e2e-predictor] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['version'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `version` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] # verify the required parameter 'namespace' is set [e2e-predictor] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['namespace'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `namespace` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] # verify the required parameter 'plural' is set [e2e-predictor] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['plural'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `plural` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] [e2e-predictor] collection_formats = {} [e2e-predictor] [e2e-predictor] path_params = {} [e2e-predictor] if 'group' in local_var_params: [e2e-predictor] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-predictor] if 'version' in local_var_params: [e2e-predictor] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-predictor] if 'namespace' in local_var_params: [e2e-predictor] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-predictor] if 'plural' in local_var_params: [e2e-predictor] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-predictor] [e2e-predictor] query_params = [] [e2e-predictor] if 'pretty' in local_var_params and local_var_params['pretty'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('pretty', local_var_params['pretty'])) # noqa: E501 [e2e-predictor] if 'allow_watch_bookmarks' in local_var_params and local_var_params['allow_watch_bookmarks'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('allowWatchBookmarks', local_var_params['allow_watch_bookmarks'])) # noqa: E501 [e2e-predictor] if '_continue' in local_var_params and local_var_params['_continue'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('continue', local_var_params['_continue'])) # noqa: E501 [e2e-predictor] if 'field_selector' in local_var_params and local_var_params['field_selector'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('fieldSelector', local_var_params['field_selector'])) # noqa: E501 [e2e-predictor] if 'label_selector' in local_var_params and local_var_params['label_selector'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('labelSelector', local_var_params['label_selector'])) # noqa: E501 [e2e-predictor] if 'limit' in local_var_params and local_var_params['limit'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('limit', local_var_params['limit'])) # noqa: E501 [e2e-predictor] if 'resource_version' in local_var_params and local_var_params['resource_version'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('resourceVersion', local_var_params['resource_version'])) # noqa: E501 [e2e-predictor] if 'resource_version_match' in local_var_params and local_var_params['resource_version_match'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('resourceVersionMatch', local_var_params['resource_version_match'])) # noqa: E501 [e2e-predictor] if 'timeout_seconds' in local_var_params and local_var_params['timeout_seconds'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('timeoutSeconds', local_var_params['timeout_seconds'])) # noqa: E501 [e2e-predictor] if 'watch' in local_var_params and local_var_params['watch'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('watch', local_var_params['watch'])) # noqa: E501 [e2e-predictor] [e2e-predictor] header_params = {} [e2e-predictor] [e2e-predictor] form_params = [] [e2e-predictor] local_var_files = {} [e2e-predictor] [e2e-predictor] body_params = None [e2e-predictor] # HTTP header `Accept` [e2e-predictor] header_params['Accept'] = self.api_client.select_header_accept( [e2e-predictor] ['application/json', 'application/json;stream=watch']) # noqa: E501 [e2e-predictor] [e2e-predictor] # Authentication setting [e2e-predictor] auth_settings = ['BearerToken'] # noqa: E501 [e2e-predictor] [e2e-predictor] > return self.api_client.call_api( [e2e-predictor] '/apis/{group}/{version}/namespaces/{namespace}/{plural}', 'GET', [e2e-predictor] path_params, [e2e-predictor] query_params, [e2e-predictor] header_params, [e2e-predictor] body=body_params, [e2e-predictor] post_params=form_params, [e2e-predictor] files=local_var_files, [e2e-predictor] response_type='object', # noqa: E501 [e2e-predictor] auth_settings=auth_settings, [e2e-predictor] async_req=local_var_params.get('async_req'), [e2e-predictor] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-predictor] _preload_content=local_var_params.get('_preload_content', True), [e2e-predictor] _request_timeout=local_var_params.get('_request_timeout'), [e2e-predictor] collection_formats=collection_formats) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:2594: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}' [e2e-predictor] method = 'GET' [e2e-predictor] path_params = {'group': 'gateway.networking.k8s.io', 'namespace': 'kserve-ci-e2e-test', 'plural': 'gateways', 'version': 'v1'} [e2e-predictor] query_params = [] [e2e-predictor] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] body = None, post_params = [], files = {}, response_type = 'object' [e2e-predictor] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-predictor] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-predictor] _host = None [e2e-predictor] [e2e-predictor] def call_api(self, resource_path, method, [e2e-predictor] path_params=None, query_params=None, header_params=None, [e2e-predictor] body=None, post_params=None, files=None, [e2e-predictor] response_type=None, auth_settings=None, async_req=None, [e2e-predictor] _return_http_data_only=None, collection_formats=None, [e2e-predictor] _preload_content=True, _request_timeout=None, _host=None): [e2e-predictor] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-predictor] [e2e-predictor] To make an async_req request, set the async_req parameter. [e2e-predictor] [e2e-predictor] :param resource_path: Path to method endpoint. [e2e-predictor] :param method: Method to call. [e2e-predictor] :param path_params: Path parameters in the url. [e2e-predictor] :param query_params: Query parameters in the url. [e2e-predictor] :param header_params: Header parameters to be [e2e-predictor] placed in the request header. [e2e-predictor] :param body: Request body. [e2e-predictor] :param post_params dict: Request post form parameters, [e2e-predictor] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-predictor] :param auth_settings list: Auth Settings names for the request. [e2e-predictor] :param response: Response data type. [e2e-predictor] :param files dict: key -> filename, value -> filepath, [e2e-predictor] for `multipart/form-data`. [e2e-predictor] :param async_req bool: execute request asynchronously [e2e-predictor] :param _return_http_data_only: response data without head status code [e2e-predictor] and headers [e2e-predictor] :param collection_formats: dict of collection formats for path, query, [e2e-predictor] header, and post parameters. [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] :return: [e2e-predictor] If async_req parameter is True, [e2e-predictor] the request will be called asynchronously. [e2e-predictor] The method will return the request thread. [e2e-predictor] If parameter async_req is False or missing, [e2e-predictor] then the method will return the response directly. [e2e-predictor] """ [e2e-predictor] if not async_req: [e2e-predictor] > return self.__call_api(resource_path, method, [e2e-predictor] path_params, query_params, header_params, [e2e-predictor] body, post_params, files, [e2e-predictor] response_type, auth_settings, [e2e-predictor] _return_http_data_only, collection_formats, [e2e-predictor] _preload_content, _request_timeout, _host) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] resource_path = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] method = 'GET' [e2e-predictor] path_params = [('group', 'gateway.networking.k8s.io'), ('version', 'v1'), ('namespace', 'kserve-ci-e2e-test'), ('plural', 'gateways')] [e2e-predictor] query_params = [] [e2e-predictor] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] body = None, post_params = [], files = {}, response_type = 'object' [e2e-predictor] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-predictor] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-predictor] _host = None [e2e-predictor] [e2e-predictor] def __call_api( [e2e-predictor] self, resource_path, method, path_params=None, [e2e-predictor] query_params=None, header_params=None, body=None, post_params=None, [e2e-predictor] files=None, response_type=None, auth_settings=None, [e2e-predictor] _return_http_data_only=None, collection_formats=None, [e2e-predictor] _preload_content=True, _request_timeout=None, _host=None): [e2e-predictor] [e2e-predictor] config = self.configuration [e2e-predictor] [e2e-predictor] # header parameters [e2e-predictor] header_params = header_params or {} [e2e-predictor] header_params.update(self.default_headers) [e2e-predictor] if self.cookie: [e2e-predictor] header_params['Cookie'] = self.cookie [e2e-predictor] if header_params: [e2e-predictor] header_params = self.sanitize_for_serialization(header_params) [e2e-predictor] header_params = dict(self.parameters_to_tuples(header_params, [e2e-predictor] collection_formats)) [e2e-predictor] [e2e-predictor] # path parameters [e2e-predictor] if path_params: [e2e-predictor] path_params = self.sanitize_for_serialization(path_params) [e2e-predictor] path_params = self.parameters_to_tuples(path_params, [e2e-predictor] collection_formats) [e2e-predictor] for k, v in path_params: [e2e-predictor] # specified safe chars, encode everything [e2e-predictor] resource_path = resource_path.replace( [e2e-predictor] '{%s}' % k, [e2e-predictor] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # query parameters [e2e-predictor] if query_params: [e2e-predictor] query_params = self.sanitize_for_serialization(query_params) [e2e-predictor] query_params = self.parameters_to_tuples(query_params, [e2e-predictor] collection_formats) [e2e-predictor] [e2e-predictor] # post parameters [e2e-predictor] if post_params or files: [e2e-predictor] post_params = post_params if post_params else [] [e2e-predictor] post_params = self.sanitize_for_serialization(post_params) [e2e-predictor] post_params = self.parameters_to_tuples(post_params, [e2e-predictor] collection_formats) [e2e-predictor] post_params.extend(self.files_parameters(files)) [e2e-predictor] [e2e-predictor] # auth setting [e2e-predictor] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-predictor] [e2e-predictor] # body [e2e-predictor] if body: [e2e-predictor] body = self.sanitize_for_serialization(body) [e2e-predictor] [e2e-predictor] # request url [e2e-predictor] if _host is None: [e2e-predictor] url = self.configuration.host + resource_path [e2e-predictor] else: [e2e-predictor] # use server/host defined in path or operation instead [e2e-predictor] url = _host + resource_path [e2e-predictor] [e2e-predictor] # perform request and return response [e2e-predictor] > response_data = self.request( [e2e-predictor] method, url, query_params=query_params, headers=header_params, [e2e-predictor] post_params=post_params, body=body, [e2e-predictor] _preload_content=_preload_content, [e2e-predictor] _request_timeout=_request_timeout) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] query_params = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] post_params = [], body = None, _preload_content = True, _request_timeout = None [e2e-predictor] [e2e-predictor] def request(self, method, url, query_params=None, headers=None, [e2e-predictor] post_params=None, body=None, _preload_content=True, [e2e-predictor] _request_timeout=None): [e2e-predictor] """Makes the HTTP request using RESTClient.""" [e2e-predictor] if method == "GET": [e2e-predictor] > return self.rest_client.GET(url, [e2e-predictor] query_params=query_params, [e2e-predictor] _preload_content=_preload_content, [e2e-predictor] _request_timeout=_request_timeout, [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:373: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] query_params = [], _preload_content = True, _request_timeout = None [e2e-predictor] [e2e-predictor] def GET(self, url, headers=None, query_params=None, _preload_content=True, [e2e-predictor] _request_timeout=None): [e2e-predictor] > return self.request("GET", url, [e2e-predictor] headers=headers, [e2e-predictor] _preload_content=_preload_content, [e2e-predictor] _request_timeout=_request_timeout, [e2e-predictor] query_params=query_params) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:244: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] query_params = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] body = None, post_params = {}, _preload_content = True, _request_timeout = None [e2e-predictor] [e2e-predictor] def request(self, method, url, query_params=None, headers=None, [e2e-predictor] body=None, post_params=None, _preload_content=True, [e2e-predictor] _request_timeout=None): [e2e-predictor] """Perform requests. [e2e-predictor] [e2e-predictor] :param method: http request method [e2e-predictor] :param url: http request url [e2e-predictor] :param query_params: query parameters in the url [e2e-predictor] :param headers: http request headers [e2e-predictor] :param body: request json body, for `application/json` [e2e-predictor] :param post_params: request post parameters, [e2e-predictor] `application/x-www-form-urlencoded` [e2e-predictor] and `multipart/form-data` [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] """ [e2e-predictor] method = method.upper() [e2e-predictor] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-predictor] 'PATCH', 'OPTIONS'] [e2e-predictor] [e2e-predictor] if post_params and body: [e2e-predictor] raise ApiValueError( [e2e-predictor] "body parameter cannot be used with post_params parameter." [e2e-predictor] ) [e2e-predictor] [e2e-predictor] post_params = post_params or {} [e2e-predictor] headers = headers or {} [e2e-predictor] [e2e-predictor] timeout = None [e2e-predictor] if _request_timeout: [e2e-predictor] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-predictor] timeout = urllib3.Timeout(total=_request_timeout) [e2e-predictor] elif (isinstance(_request_timeout, tuple) and [e2e-predictor] len(_request_timeout) == 2): [e2e-predictor] timeout = urllib3.Timeout( [e2e-predictor] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-predictor] [e2e-predictor] if 'Content-Type' not in headers: [e2e-predictor] headers['Content-Type'] = 'application/json' [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-predictor] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-predictor] if query_params: [e2e-predictor] url += '?' + urlencode(query_params) [e2e-predictor] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-predictor] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-predictor] if headers['Content-Type'] == 'application/json-patch+json': [e2e-predictor] if not isinstance(body, list): [e2e-predictor] headers['Content-Type'] = \ [e2e-predictor] 'application/strategic-merge-patch+json' [e2e-predictor] request_body = None [e2e-predictor] if body is not None: [e2e-predictor] request_body = json.dumps(body) [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] body=request_body, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] fields=post_params, [e2e-predictor] encode_multipart=False, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] elif headers['Content-Type'] == 'multipart/form-data': [e2e-predictor] # must del headers['Content-Type'], or the correct [e2e-predictor] # Content-Type which generated by urllib3 will be [e2e-predictor] # overwritten. [e2e-predictor] del headers['Content-Type'] [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] fields=post_params, [e2e-predictor] encode_multipart=True, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] # Pass a `string` parameter directly in the body to support [e2e-predictor] # other content types than Json when `body` argument is [e2e-predictor] # provided in serialized form [e2e-predictor] elif isinstance(body, str) or isinstance(body, bytes): [e2e-predictor] request_body = body [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] body=request_body, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] else: [e2e-predictor] # Cannot generate the request from given parameters [e2e-predictor] msg = """Cannot prepare a request message for provided [e2e-predictor] arguments. Please check that your arguments match [e2e-predictor] declared content type.""" [e2e-predictor] raise ApiException(status=0, reason=msg) [e2e-predictor] # For `GET`, `HEAD` [e2e-predictor] else: [e2e-predictor] > r = self.pool_manager.request(method, url, [e2e-predictor] fields=query_params, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:217: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None, fields = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] json = None, urlopen_kw = {'preload_content': True, 'timeout': None} [e2e-predictor] [e2e-predictor] def request( [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] fields: _TYPE_FIELDS | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] json: typing.Any | None = None, [e2e-predictor] **urlopen_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Make a request using :meth:`urlopen` with the appropriate encoding of [e2e-predictor] ``fields`` based on the ``method`` used. [e2e-predictor] [e2e-predictor] This is a convenience method that requires the least amount of manual [e2e-predictor] effort. It can be used in most situations, while still having the [e2e-predictor] option to drop down to more specific methods when necessary, such as [e2e-predictor] :meth:`request_encode_url`, :meth:`request_encode_body`, [e2e-predictor] or even the lowest level :meth:`urlopen`. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param fields: [e2e-predictor] Data to encode and send in the URL or request body, depending on ``method``. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param json: [e2e-predictor] Data to encode and send as JSON with UTF-encoded in the request body. [e2e-predictor] The ``"Content-Type"`` header will be set to ``"application/json"`` [e2e-predictor] unless specified otherwise. [e2e-predictor] """ [e2e-predictor] method = method.upper() [e2e-predictor] [e2e-predictor] if json is not None and body is not None: [e2e-predictor] raise TypeError( [e2e-predictor] "request got values for both 'body' and 'json' parameters which are mutually exclusive" [e2e-predictor] ) [e2e-predictor] [e2e-predictor] if json is not None: [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not ("content-type" in map(str.lower, headers.keys())): [e2e-predictor] headers = HTTPHeaderDict(headers) [e2e-predictor] headers["Content-Type"] = "application/json" [e2e-predictor] [e2e-predictor] body = _json.dumps(json, separators=(",", ":"), ensure_ascii=False).encode( [e2e-predictor] "utf-8" [e2e-predictor] ) [e2e-predictor] [e2e-predictor] if body is not None: [e2e-predictor] urlopen_kw["body"] = body [e2e-predictor] [e2e-predictor] if method in self._encode_url_methods: [e2e-predictor] > return self.request_encode_url( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] fields=fields, # type: ignore[arg-type] [e2e-predictor] headers=headers, [e2e-predictor] **urlopen_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/_request_methods.py:135: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] fields = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] urlopen_kw = {'preload_content': True, 'timeout': None} [e2e-predictor] extra_kw = {'headers': {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'}, 'preload_content': True, 'timeout': None} [e2e-predictor] [e2e-predictor] def request_encode_url( [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] fields: _TYPE_ENCODE_URL_FIELDS | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] **urlopen_kw: str, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Make a request using :meth:`urlopen` with the ``fields`` encoded in [e2e-predictor] the url. This is useful for request methods like GET, HEAD, DELETE, etc. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param fields: [e2e-predictor] Data to encode and send in the URL. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] """ [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] extra_kw: dict[str, typing.Any] = {"headers": headers} [e2e-predictor] extra_kw.update(urlopen_kw) [e2e-predictor] [e2e-predictor] if fields: [e2e-predictor] url += "?" + urlencode(fields) [e2e-predictor] [e2e-predictor] > return self.urlopen(method, url, **extra_kw) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/_request_methods.py:182: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] redirect = True [e2e-predictor] kw = {'assert_same_host': False, 'headers': {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'}, 'preload_content': True, 'redirect': False, ...} [e2e-predictor] u = Url(scheme='https', auth=None, host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443, path='/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways', query=None, fragment=None) [e2e-predictor] conn = [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, method: str, url: str, redirect: bool = True, **kw: typing.Any [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Same as :meth:`urllib3.HTTPConnectionPool.urlopen` [e2e-predictor] with custom cross-host redirect logic and only sends the request-uri [e2e-predictor] portion of the ``url``. [e2e-predictor] [e2e-predictor] The given ``url`` parameter must be absolute, such that an appropriate [e2e-predictor] :class:`urllib3.connectionpool.ConnectionPool` can be chosen for it. [e2e-predictor] """ [e2e-predictor] u = parse_url(url) [e2e-predictor] [e2e-predictor] if u.scheme is None: [e2e-predictor] warnings.warn( [e2e-predictor] "URLs without a scheme (ie 'https://') are deprecated and will raise an error " [e2e-predictor] "in urllib3 v3.0. To avoid this FutureWarning ensure all URLs " [e2e-predictor] "start with 'https://' or 'http://'. Read more in this issue: " [e2e-predictor] "https://github.com/urllib3/urllib3/issues/2920", [e2e-predictor] category=FutureWarning, [e2e-predictor] stacklevel=2, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme) [e2e-predictor] [e2e-predictor] kw["assert_same_host"] = False [e2e-predictor] kw["redirect"] = False [e2e-predictor] [e2e-predictor] if "headers" not in kw: [e2e-predictor] kw["headers"] = self.headers [e2e-predictor] [e2e-predictor] if self._proxy_requires_url_absolute_form(u): [e2e-predictor] response = conn.urlopen(method, url, **kw) [e2e-predictor] else: [e2e-predictor] > response = conn.urlopen(method, u.request_uri, **kw) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/poolmanager.py:457: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=2, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False [e2e-predictor] err = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] retries.sleep() [e2e-predictor] [e2e-predictor] # Keep track of the error for the retry warning. [e2e-predictor] err = e [e2e-predictor] [e2e-predictor] finally: [e2e-predictor] if not clean_exit: [e2e-predictor] # We hit some kind of exception, handled or otherwise. We need [e2e-predictor] # to throw the connection away unless explicitly told not to. [e2e-predictor] # Close the connection, set the variable to None, and make sure [e2e-predictor] # we put the None back in the pool to avoid leaking it. [e2e-predictor] if conn: [e2e-predictor] conn.close() [e2e-predictor] conn = None [e2e-predictor] release_this_conn = True [e2e-predictor] [e2e-predictor] if release_this_conn: [e2e-predictor] # Put the connection back to be reused. If the connection is [e2e-predictor] # expired then it will be None, which will get replaced with a [e2e-predictor] # fresh connection during _get_conn. [e2e-predictor] self._put_conn(conn) [e2e-predictor] [e2e-predictor] if not conn: [e2e-predictor] # Try again [e2e-predictor] log.warning( [e2e-predictor] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-predictor] ) [e2e-predictor] > return self.urlopen( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] body, [e2e-predictor] headers, [e2e-predictor] retries, [e2e-predictor] redirect, [e2e-predictor] assert_same_host, [e2e-predictor] timeout=timeout, [e2e-predictor] pool_timeout=pool_timeout, [e2e-predictor] release_conn=release_conn, [e2e-predictor] chunked=chunked, [e2e-predictor] body_pos=body_pos, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:872: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=1, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False [e2e-predictor] err = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] retries.sleep() [e2e-predictor] [e2e-predictor] # Keep track of the error for the retry warning. [e2e-predictor] err = e [e2e-predictor] [e2e-predictor] finally: [e2e-predictor] if not clean_exit: [e2e-predictor] # We hit some kind of exception, handled or otherwise. We need [e2e-predictor] # to throw the connection away unless explicitly told not to. [e2e-predictor] # Close the connection, set the variable to None, and make sure [e2e-predictor] # we put the None back in the pool to avoid leaking it. [e2e-predictor] if conn: [e2e-predictor] conn.close() [e2e-predictor] conn = None [e2e-predictor] release_this_conn = True [e2e-predictor] [e2e-predictor] if release_this_conn: [e2e-predictor] # Put the connection back to be reused. If the connection is [e2e-predictor] # expired then it will be None, which will get replaced with a [e2e-predictor] # fresh connection during _get_conn. [e2e-predictor] self._put_conn(conn) [e2e-predictor] [e2e-predictor] if not conn: [e2e-predictor] # Try again [e2e-predictor] log.warning( [e2e-predictor] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-predictor] ) [e2e-predictor] > return self.urlopen( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] body, [e2e-predictor] headers, [e2e-predictor] retries, [e2e-predictor] redirect, [e2e-predictor] assert_same_host, [e2e-predictor] timeout=timeout, [e2e-predictor] pool_timeout=pool_timeout, [e2e-predictor] release_conn=release_conn, [e2e-predictor] chunked=chunked, [e2e-predictor] body_pos=body_pos, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:872: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False [e2e-predictor] err = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] retries.sleep() [e2e-predictor] [e2e-predictor] # Keep track of the error for the retry warning. [e2e-predictor] err = e [e2e-predictor] [e2e-predictor] finally: [e2e-predictor] if not clean_exit: [e2e-predictor] # We hit some kind of exception, handled or otherwise. We need [e2e-predictor] # to throw the connection away unless explicitly told not to. [e2e-predictor] # Close the connection, set the variable to None, and make sure [e2e-predictor] # we put the None back in the pool to avoid leaking it. [e2e-predictor] if conn: [e2e-predictor] conn.close() [e2e-predictor] conn = None [e2e-predictor] release_this_conn = True [e2e-predictor] [e2e-predictor] if release_this_conn: [e2e-predictor] # Put the connection back to be reused. If the connection is [e2e-predictor] # expired then it will be None, which will get replaced with a [e2e-predictor] # fresh connection during _get_conn. [e2e-predictor] self._put_conn(conn) [e2e-predictor] [e2e-predictor] if not conn: [e2e-predictor] # Try again [e2e-predictor] log.warning( [e2e-predictor] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-predictor] ) [e2e-predictor] > return self.urlopen( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] body, [e2e-predictor] headers, [e2e-predictor] retries, [e2e-predictor] redirect, [e2e-predictor] assert_same_host, [e2e-predictor] timeout=timeout, [e2e-predictor] pool_timeout=pool_timeout, [e2e-predictor] release_conn=release_conn, [e2e-predictor] chunked=chunked, [e2e-predictor] body_pos=body_pos, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:872: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False, err = None [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] > retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:842: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] response = None [e2e-predictor] error = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] _pool = [e2e-predictor] _stacktrace = [e2e-predictor] [e2e-predictor] def increment( [e2e-predictor] self, [e2e-predictor] method: str | None = None, [e2e-predictor] url: str | None = None, [e2e-predictor] response: BaseHTTPResponse | None = None, [e2e-predictor] error: Exception | None = None, [e2e-predictor] _pool: ConnectionPool | None = None, [e2e-predictor] _stacktrace: TracebackType | None = None, [e2e-predictor] ) -> Self: [e2e-predictor] """Return a new Retry object with incremented retry counters. [e2e-predictor] [e2e-predictor] :param response: A response object, or None, if the server did not [e2e-predictor] return a response. [e2e-predictor] :type response: :class:`~urllib3.response.BaseHTTPResponse` [e2e-predictor] :param Exception error: An error encountered during the request, or [e2e-predictor] None if the response was received successfully. [e2e-predictor] [e2e-predictor] :return: A new ``Retry`` object. [e2e-predictor] """ [e2e-predictor] if self.total is False and error: [e2e-predictor] # Disabled, indicate to re-raise the error. [e2e-predictor] raise reraise(type(error), error, _stacktrace) [e2e-predictor] [e2e-predictor] total = self.total [e2e-predictor] if total is not None: [e2e-predictor] total -= 1 [e2e-predictor] [e2e-predictor] connect = self.connect [e2e-predictor] read = self.read [e2e-predictor] redirect = self.redirect [e2e-predictor] status_count = self.status [e2e-predictor] other = self.other [e2e-predictor] cause = "unknown" [e2e-predictor] status = None [e2e-predictor] redirect_location = None [e2e-predictor] [e2e-predictor] if error and self._is_connection_error(error): [e2e-predictor] # Connect retry? [e2e-predictor] if connect is False: [e2e-predictor] raise reraise(type(error), error, _stacktrace) [e2e-predictor] elif connect is not None: [e2e-predictor] connect -= 1 [e2e-predictor] [e2e-predictor] elif error and self._is_read_error(error): [e2e-predictor] # Read retry? [e2e-predictor] if read is False or method is None or not self._is_method_retryable(method): [e2e-predictor] raise reraise(type(error), error, _stacktrace) [e2e-predictor] elif read is not None: [e2e-predictor] read -= 1 [e2e-predictor] [e2e-predictor] elif error: [e2e-predictor] # Other retry? [e2e-predictor] if other is not None: [e2e-predictor] other -= 1 [e2e-predictor] [e2e-predictor] elif response and response.get_redirect_location(): [e2e-predictor] # Redirect retry? [e2e-predictor] if redirect is not None: [e2e-predictor] redirect -= 1 [e2e-predictor] cause = "too many redirects" [e2e-predictor] response_redirect_location = response.get_redirect_location() [e2e-predictor] if response_redirect_location: [e2e-predictor] redirect_location = response_redirect_location [e2e-predictor] status = response.status [e2e-predictor] [e2e-predictor] else: [e2e-predictor] # Incrementing because of a server error like a 500 in [e2e-predictor] # status_forcelist and the given method is in the allowed_methods [e2e-predictor] cause = ResponseError.GENERIC_ERROR [e2e-predictor] if response and response.status: [e2e-predictor] if status_count is not None: [e2e-predictor] status_count -= 1 [e2e-predictor] cause = ResponseError.SPECIFIC_ERROR.format(status_code=response.status) [e2e-predictor] status = response.status [e2e-predictor] [e2e-predictor] history = self.history + ( [e2e-predictor] RequestHistory(method, url, error, status, redirect_location), [e2e-predictor] ) [e2e-predictor] [e2e-predictor] new_retry = self.new( [e2e-predictor] total=total, [e2e-predictor] connect=connect, [e2e-predictor] read=read, [e2e-predictor] redirect=redirect, [e2e-predictor] status=status_count, [e2e-predictor] other=other, [e2e-predictor] history=history, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] if new_retry.is_exhausted(): [e2e-predictor] reason = error or ResponseError(cause) [e2e-predictor] > raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] [e2e-predictor] E urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Max retries exceeded with url: /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways (Caused by NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/retry.py:543: MaxRetryError [e2e-predictor] ------------------------------ Captured log setup ------------------------------ [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways [e2e-predictor] ___________________ ERROR at setup of test_sklearn_v2_mixed ____________________ [e2e-predictor] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-predictor] [e2e-predictor] self = [e2e-predictor] [e2e-predictor] def _new_conn(self) -> socket.socket: [e2e-predictor] """Establish a socket connection and set nodelay settings on it. [e2e-predictor] [e2e-predictor] :return: New socket connection. [e2e-predictor] """ [e2e-predictor] try: [e2e-predictor] > sock = connection.create_connection( [e2e-predictor] (self._dns_host, self.port), [e2e-predictor] self.timeout, [e2e-predictor] source_address=self.source_address, [e2e-predictor] socket_options=self.socket_options, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:204: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] address = ('a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', 6443) [e2e-predictor] timeout = None, source_address = None, socket_options = [(6, 1, 1)] [e2e-predictor] [e2e-predictor] def create_connection( [e2e-predictor] address: tuple[str, int], [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] source_address: tuple[str, int] | None = None, [e2e-predictor] socket_options: _TYPE_SOCKET_OPTIONS | None = None, [e2e-predictor] ) -> socket.socket: [e2e-predictor] """Connect to *address* and return the socket object. [e2e-predictor] [e2e-predictor] Convenience function. Connect to *address* (a 2-tuple ``(host, [e2e-predictor] port)``) and return the socket object. Passing the optional [e2e-predictor] *timeout* parameter will set the timeout on the socket instance [e2e-predictor] before attempting to connect. If no *timeout* is supplied, the [e2e-predictor] global default timeout setting returned by :func:`socket.getdefaulttimeout` [e2e-predictor] is used. If *source_address* is set it must be a tuple of (host, port) [e2e-predictor] for the socket to bind as a source address before making the connection. [e2e-predictor] An host of '' or port 0 tells the OS to use the default. [e2e-predictor] """ [e2e-predictor] [e2e-predictor] host, port = address [e2e-predictor] if host.startswith("["): [e2e-predictor] host = host.strip("[]") [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Using the value from allowed_gai_family() in the context of getaddrinfo lets [e2e-predictor] # us select whether to work with IPv4 DNS records, IPv6 records, or both. [e2e-predictor] # The original create_connection function always returns all records. [e2e-predictor] family = allowed_gai_family() [e2e-predictor] [e2e-predictor] try: [e2e-predictor] host.encode("idna") [e2e-predictor] except UnicodeError: [e2e-predictor] raise LocationParseError(f"'{host}', label empty or too long") from None [e2e-predictor] [e2e-predictor] > for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/connection.py:60: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] host = 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' [e2e-predictor] port = 6443, family = [e2e-predictor] type = , proto = 0, flags = 0 [e2e-predictor] [e2e-predictor] def getaddrinfo(host, port, family=0, type=0, proto=0, flags=0): [e2e-predictor] """Resolve host and port into list of address info entries. [e2e-predictor] [e2e-predictor] Translate the host/port argument into a sequence of 5-tuples that contain [e2e-predictor] all the necessary arguments for creating a socket connected to that service. [e2e-predictor] host is a domain name, a string representation of an IPv4/v6 address or [e2e-predictor] None. port is a string service name such as 'http', a numeric port number or [e2e-predictor] None. By passing None as the value of host and port, you can pass NULL to [e2e-predictor] the underlying C API. [e2e-predictor] [e2e-predictor] The family, type and proto arguments can be optionally specified in order to [e2e-predictor] narrow the list of addresses returned. Passing zero as a value for each of [e2e-predictor] these arguments selects the full range of results. [e2e-predictor] """ [e2e-predictor] # We override this function since we want to translate the numeric family [e2e-predictor] # and socket type values to enum constants. [e2e-predictor] addrlist = [] [e2e-predictor] > for res in _socket.getaddrinfo(host, port, family, type, proto, flags): [e2e-predictor] E socket.gaierror: [Errno -2] Name or service not known [e2e-predictor] [e2e-predictor] /usr/lib64/python3.11/socket.py:974: gaierror [e2e-predictor] [e2e-predictor] The above exception was the direct cause of the following exception: [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False, err = None [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] > response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:788: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] conn = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] timeout = Timeout(connect=None, read=None, total=None), chunked = False [e2e-predictor] response_conn = None, preload_content = True, decode_content = True [e2e-predictor] enforce_content_length = True [e2e-predictor] [e2e-predictor] def _make_request( [e2e-predictor] self, [e2e-predictor] conn: BaseHTTPConnection, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | None = None, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] chunked: bool = False, [e2e-predictor] response_conn: BaseHTTPConnection | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] enforce_content_length: bool = True, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Perform a request on a given urllib connection object taken from our [e2e-predictor] pool. [e2e-predictor] [e2e-predictor] :param conn: [e2e-predictor] a connection from one of our connection pools [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] Pass ``None`` to retry until you receive a response. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param response_conn: [e2e-predictor] Set this to ``None`` if you will handle releasing the connection or [e2e-predictor] set the connection to have the response release it. [e2e-predictor] [e2e-predictor] :param preload_content: [e2e-predictor] If True, the response's body will be preloaded during construction. [e2e-predictor] [e2e-predictor] :param decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param enforce_content_length: [e2e-predictor] Enforce content length checking. Body returned by server must match [e2e-predictor] value of Content-Length header, if present. Otherwise, raise error. [e2e-predictor] """ [e2e-predictor] self.num_requests += 1 [e2e-predictor] [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] timeout_obj.start_connect() [e2e-predictor] conn.timeout = Timeout.resolve_default_timeout(timeout_obj.connect_timeout) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Trigger any extra validation we need to do. [e2e-predictor] try: [e2e-predictor] self._validate_conn(conn) [e2e-predictor] except (SocketTimeout, BaseSSLError) as e: [e2e-predictor] self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # _validate_conn() starts the connection to an HTTPS proxy [e2e-predictor] # so we need to wrap errors with 'ProxyError' here too. [e2e-predictor] except ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] BaseSSLError, [e2e-predictor] CertificateError, [e2e-predictor] SSLError, [e2e-predictor] ) as e: [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] # If the connection didn't successfully connect to it's proxy [e2e-predictor] # then there [e2e-predictor] if isinstance( [e2e-predictor] new_e, (OSError, NewConnectionError, TimeoutError, SSLError) [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] > raise new_e [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:488: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] conn = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] timeout = Timeout(connect=None, read=None, total=None), chunked = False [e2e-predictor] response_conn = None, preload_content = True, decode_content = True [e2e-predictor] enforce_content_length = True [e2e-predictor] [e2e-predictor] def _make_request( [e2e-predictor] self, [e2e-predictor] conn: BaseHTTPConnection, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | None = None, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] chunked: bool = False, [e2e-predictor] response_conn: BaseHTTPConnection | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] enforce_content_length: bool = True, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Perform a request on a given urllib connection object taken from our [e2e-predictor] pool. [e2e-predictor] [e2e-predictor] :param conn: [e2e-predictor] a connection from one of our connection pools [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] Pass ``None`` to retry until you receive a response. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param response_conn: [e2e-predictor] Set this to ``None`` if you will handle releasing the connection or [e2e-predictor] set the connection to have the response release it. [e2e-predictor] [e2e-predictor] :param preload_content: [e2e-predictor] If True, the response's body will be preloaded during construction. [e2e-predictor] [e2e-predictor] :param decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param enforce_content_length: [e2e-predictor] Enforce content length checking. Body returned by server must match [e2e-predictor] value of Content-Length header, if present. Otherwise, raise error. [e2e-predictor] """ [e2e-predictor] self.num_requests += 1 [e2e-predictor] [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] timeout_obj.start_connect() [e2e-predictor] conn.timeout = Timeout.resolve_default_timeout(timeout_obj.connect_timeout) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Trigger any extra validation we need to do. [e2e-predictor] try: [e2e-predictor] > self._validate_conn(conn) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:464: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] conn = [e2e-predictor] [e2e-predictor] def _validate_conn(self, conn: BaseHTTPConnection) -> None: [e2e-predictor] """ [e2e-predictor] Called right before a request is made, after the socket is created. [e2e-predictor] """ [e2e-predictor] super()._validate_conn(conn) [e2e-predictor] [e2e-predictor] # Force connect early to allow us to validate the connection. [e2e-predictor] if conn.is_closed: [e2e-predictor] > conn.connect() [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:1106: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] [e2e-predictor] def connect(self) -> None: [e2e-predictor] # Today we don't need to be doing this step before the /actual/ socket [e2e-predictor] # connection, however in the future we'll need to decide whether to [e2e-predictor] # create a new socket or re-use an existing "shared" socket as a part [e2e-predictor] # of the HTTP/2 handshake dance. [e2e-predictor] if self._tunnel_host is not None and self._tunnel_port is not None: [e2e-predictor] probe_http2_host = self._tunnel_host [e2e-predictor] probe_http2_port = self._tunnel_port [e2e-predictor] else: [e2e-predictor] probe_http2_host = self.host [e2e-predictor] probe_http2_port = self.port [e2e-predictor] [e2e-predictor] # Check if the target origin supports HTTP/2. [e2e-predictor] # If the value comes back as 'None' it means that the current thread [e2e-predictor] # is probing for HTTP/2 support. Otherwise, we're waiting for another [e2e-predictor] # probe to complete, or we get a value right away. [e2e-predictor] target_supports_http2: bool | None [e2e-predictor] if "h2" in ssl_.ALPN_PROTOCOLS: [e2e-predictor] target_supports_http2 = http2_probe.acquire_and_get( [e2e-predictor] host=probe_http2_host, port=probe_http2_port [e2e-predictor] ) [e2e-predictor] else: [e2e-predictor] # If HTTP/2 isn't going to be offered it doesn't matter if [e2e-predictor] # the target supports HTTP/2. Don't want to make a probe. [e2e-predictor] target_supports_http2 = False [e2e-predictor] [e2e-predictor] if self._connect_callback is not None: [e2e-predictor] self._connect_callback( [e2e-predictor] "before connect", [e2e-predictor] thread_id=threading.get_ident(), [e2e-predictor] target_supports_http2=target_supports_http2, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] sock: socket.socket | ssl.SSLSocket [e2e-predictor] > self.sock = sock = self._new_conn() [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:759: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] [e2e-predictor] def _new_conn(self) -> socket.socket: [e2e-predictor] """Establish a socket connection and set nodelay settings on it. [e2e-predictor] [e2e-predictor] :return: New socket connection. [e2e-predictor] """ [e2e-predictor] try: [e2e-predictor] sock = connection.create_connection( [e2e-predictor] (self._dns_host, self.port), [e2e-predictor] self.timeout, [e2e-predictor] source_address=self.source_address, [e2e-predictor] socket_options=self.socket_options, [e2e-predictor] ) [e2e-predictor] except socket.gaierror as e: [e2e-predictor] > raise NameResolutionError(self.host, self, e) from e [e2e-predictor] E urllib3.exceptions.NameResolutionError: HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:211: NameResolutionError [e2e-predictor] [e2e-predictor] The above exception was the direct cause of the following exception: [e2e-predictor] [e2e-predictor] request = > [e2e-predictor] [e2e-predictor] @pytest.fixture(autouse=True) [e2e-predictor] def ensure_gateway_proxy_memory(request): [e2e-predictor] """After test setup creates gateways, patch them for proxy memory.""" [e2e-predictor] if not GATEWAY_PROXY_MEMORY: [e2e-predictor] return [e2e-predictor] [e2e-predictor] # Let test_case (llmisvc) create gateways first [e2e-predictor] [e2e-predictor] if "test_case" in request.fixturenames: [e2e-predictor] request.getfixturevalue("test_case") [e2e-predictor] [e2e-predictor] from kserve import KServeClient [e2e-predictor] [e2e-predictor] kserve_client = KServeClient( [e2e-predictor] config_file=os.environ.get("KUBECONFIG", "~/.kube/config") [e2e-predictor] ) [e2e-predictor] api_client = kserve_client.api_instance.api_client [e2e-predictor] [e2e-predictor] namespace = os.environ.get("KSERVE_TEST_NAMESPACE", "kserve-ci-e2e-test") [e2e-predictor] _ensure_configmap(namespace, api_client) [e2e-predictor] > patched = _patch_gateways(namespace, api_client) [e2e-predictor] [e2e-predictor] common/gateway_proxy_istio.py:194: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] namespace = 'kserve-ci-e2e-test' [e2e-predictor] api_client = [e2e-predictor] [e2e-predictor] def _patch_gateways(namespace, api_client): [e2e-predictor] """Patch all Gateways in namespace to include parametersRef. [e2e-predictor] [e2e-predictor] Returns the list of gateway names that were actually patched. [e2e-predictor] """ [e2e-predictor] custom_api = client.CustomObjectsApi(api_client) [e2e-predictor] > gateways = custom_api.list_namespaced_custom_object( [e2e-predictor] "gateway.networking.k8s.io", "v1", namespace, "gateways" [e2e-predictor] ) [e2e-predictor] [e2e-predictor] common/gateway_proxy_istio.py:117: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] group = 'gateway.networking.k8s.io', version = 'v1' [e2e-predictor] namespace = 'kserve-ci-e2e-test', plural = 'gateways' [e2e-predictor] kwargs = {'_return_http_data_only': True} [e2e-predictor] [e2e-predictor] def list_namespaced_custom_object(self, group, version, namespace, plural, **kwargs): # noqa: E501 [e2e-predictor] """list_namespaced_custom_object # noqa: E501 [e2e-predictor] [e2e-predictor] list or watch namespace scoped custom objects # noqa: E501 [e2e-predictor] This method makes a synchronous HTTP request by default. To make an [e2e-predictor] asynchronous HTTP request, please pass async_req=True [e2e-predictor] >>> thread = api.list_namespaced_custom_object(group, version, namespace, plural, async_req=True) [e2e-predictor] >>> result = thread.get() [e2e-predictor] [e2e-predictor] :param async_req bool: execute request asynchronously [e2e-predictor] :param str group: The custom resource's group name (required) [e2e-predictor] :param str version: The custom resource's version (required) [e2e-predictor] :param str namespace: The custom resource's namespace (required) [e2e-predictor] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-predictor] :param str pretty: If 'true', then the output is pretty printed. [e2e-predictor] :param bool allow_watch_bookmarks: allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored. [e2e-predictor] :param str _continue: The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\". This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications. [e2e-predictor] :param str field_selector: A selector to restrict the list of returned objects by their fields. Defaults to everything. [e2e-predictor] :param str label_selector: A selector to restrict the list of returned objects by their labels. Defaults to everything. [e2e-predictor] :param int limit: limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true. The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned. [e2e-predictor] :param str resource_version: When specified with a watch call, shows changes that occur after that particular version of a resource. Defaults to changes from the beginning of history. When specified for list: - if unset, then the result is returned from remote storage based on quorum-read flag; - if it's 0, then we simply return what we currently have in cache, no guarantee; - if set to non zero, then the result is at least as fresh as given rv. [e2e-predictor] :param str resource_version_match: resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details. Defaults to unset [e2e-predictor] :param int timeout_seconds: Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity. [e2e-predictor] :param bool watch: Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] :return: object [e2e-predictor] If the method is called asynchronously, [e2e-predictor] returns the request thread. [e2e-predictor] """ [e2e-predictor] kwargs['_return_http_data_only'] = True [e2e-predictor] > return self.list_namespaced_custom_object_with_http_info(group, version, namespace, plural, **kwargs) # noqa: E501 [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:2455: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] group = 'gateway.networking.k8s.io', version = 'v1' [e2e-predictor] namespace = 'kserve-ci-e2e-test', plural = 'gateways' [e2e-predictor] kwargs = {'_return_http_data_only': True} [e2e-predictor] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'pretty', 'allow_watch_bookmarks', ...], 'auth_settings': ['BearerToken'], 'body_params': None, ...} [e2e-predictor] all_params = ['group', 'version', 'namespace', 'plural', 'pretty', 'allow_watch_bookmarks', ...] [e2e-predictor] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-predictor] path_params = {'group': 'gateway.networking.k8s.io', 'namespace': 'kserve-ci-e2e-test', 'plural': 'gateways', 'version': 'v1'} [e2e-predictor] query_params = [] [e2e-predictor] [e2e-predictor] def list_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, **kwargs): # noqa: E501 [e2e-predictor] """list_namespaced_custom_object # noqa: E501 [e2e-predictor] [e2e-predictor] list or watch namespace scoped custom objects # noqa: E501 [e2e-predictor] This method makes a synchronous HTTP request by default. To make an [e2e-predictor] asynchronous HTTP request, please pass async_req=True [e2e-predictor] >>> thread = api.list_namespaced_custom_object_with_http_info(group, version, namespace, plural, async_req=True) [e2e-predictor] >>> result = thread.get() [e2e-predictor] [e2e-predictor] :param async_req bool: execute request asynchronously [e2e-predictor] :param str group: The custom resource's group name (required) [e2e-predictor] :param str version: The custom resource's version (required) [e2e-predictor] :param str namespace: The custom resource's namespace (required) [e2e-predictor] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-predictor] :param str pretty: If 'true', then the output is pretty printed. [e2e-predictor] :param bool allow_watch_bookmarks: allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored. [e2e-predictor] :param str _continue: The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\". This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications. [e2e-predictor] :param str field_selector: A selector to restrict the list of returned objects by their fields. Defaults to everything. [e2e-predictor] :param str label_selector: A selector to restrict the list of returned objects by their labels. Defaults to everything. [e2e-predictor] :param int limit: limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true. The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned. [e2e-predictor] :param str resource_version: When specified with a watch call, shows changes that occur after that particular version of a resource. Defaults to changes from the beginning of history. When specified for list: - if unset, then the result is returned from remote storage based on quorum-read flag; - if it's 0, then we simply return what we currently have in cache, no guarantee; - if set to non zero, then the result is at least as fresh as given rv. [e2e-predictor] :param str resource_version_match: resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details. Defaults to unset [e2e-predictor] :param int timeout_seconds: Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity. [e2e-predictor] :param bool watch: Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. [e2e-predictor] :param _return_http_data_only: response data without head status code [e2e-predictor] and headers [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-predictor] If the method is called asynchronously, [e2e-predictor] returns the request thread. [e2e-predictor] """ [e2e-predictor] [e2e-predictor] local_var_params = locals() [e2e-predictor] [e2e-predictor] all_params = [ [e2e-predictor] 'group', [e2e-predictor] 'version', [e2e-predictor] 'namespace', [e2e-predictor] 'plural', [e2e-predictor] 'pretty', [e2e-predictor] 'allow_watch_bookmarks', [e2e-predictor] '_continue', [e2e-predictor] 'field_selector', [e2e-predictor] 'label_selector', [e2e-predictor] 'limit', [e2e-predictor] 'resource_version', [e2e-predictor] 'resource_version_match', [e2e-predictor] 'timeout_seconds', [e2e-predictor] 'watch' [e2e-predictor] ] [e2e-predictor] all_params.extend( [e2e-predictor] [ [e2e-predictor] 'async_req', [e2e-predictor] '_return_http_data_only', [e2e-predictor] '_preload_content', [e2e-predictor] '_request_timeout' [e2e-predictor] ] [e2e-predictor] ) [e2e-predictor] [e2e-predictor] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-predictor] if key not in all_params: [e2e-predictor] raise ApiTypeError( [e2e-predictor] "Got an unexpected keyword argument '%s'" [e2e-predictor] " to method list_namespaced_custom_object" % key [e2e-predictor] ) [e2e-predictor] local_var_params[key] = val [e2e-predictor] del local_var_params['kwargs'] [e2e-predictor] # verify the required parameter 'group' is set [e2e-predictor] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['group'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `group` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] # verify the required parameter 'version' is set [e2e-predictor] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['version'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `version` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] # verify the required parameter 'namespace' is set [e2e-predictor] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['namespace'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `namespace` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] # verify the required parameter 'plural' is set [e2e-predictor] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['plural'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `plural` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] [e2e-predictor] collection_formats = {} [e2e-predictor] [e2e-predictor] path_params = {} [e2e-predictor] if 'group' in local_var_params: [e2e-predictor] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-predictor] if 'version' in local_var_params: [e2e-predictor] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-predictor] if 'namespace' in local_var_params: [e2e-predictor] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-predictor] if 'plural' in local_var_params: [e2e-predictor] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-predictor] [e2e-predictor] query_params = [] [e2e-predictor] if 'pretty' in local_var_params and local_var_params['pretty'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('pretty', local_var_params['pretty'])) # noqa: E501 [e2e-predictor] if 'allow_watch_bookmarks' in local_var_params and local_var_params['allow_watch_bookmarks'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('allowWatchBookmarks', local_var_params['allow_watch_bookmarks'])) # noqa: E501 [e2e-predictor] if '_continue' in local_var_params and local_var_params['_continue'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('continue', local_var_params['_continue'])) # noqa: E501 [e2e-predictor] if 'field_selector' in local_var_params and local_var_params['field_selector'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('fieldSelector', local_var_params['field_selector'])) # noqa: E501 [e2e-predictor] if 'label_selector' in local_var_params and local_var_params['label_selector'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('labelSelector', local_var_params['label_selector'])) # noqa: E501 [e2e-predictor] if 'limit' in local_var_params and local_var_params['limit'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('limit', local_var_params['limit'])) # noqa: E501 [e2e-predictor] if 'resource_version' in local_var_params and local_var_params['resource_version'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('resourceVersion', local_var_params['resource_version'])) # noqa: E501 [e2e-predictor] if 'resource_version_match' in local_var_params and local_var_params['resource_version_match'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('resourceVersionMatch', local_var_params['resource_version_match'])) # noqa: E501 [e2e-predictor] if 'timeout_seconds' in local_var_params and local_var_params['timeout_seconds'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('timeoutSeconds', local_var_params['timeout_seconds'])) # noqa: E501 [e2e-predictor] if 'watch' in local_var_params and local_var_params['watch'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('watch', local_var_params['watch'])) # noqa: E501 [e2e-predictor] [e2e-predictor] header_params = {} [e2e-predictor] [e2e-predictor] form_params = [] [e2e-predictor] local_var_files = {} [e2e-predictor] [e2e-predictor] body_params = None [e2e-predictor] # HTTP header `Accept` [e2e-predictor] header_params['Accept'] = self.api_client.select_header_accept( [e2e-predictor] ['application/json', 'application/json;stream=watch']) # noqa: E501 [e2e-predictor] [e2e-predictor] # Authentication setting [e2e-predictor] auth_settings = ['BearerToken'] # noqa: E501 [e2e-predictor] [e2e-predictor] > return self.api_client.call_api( [e2e-predictor] '/apis/{group}/{version}/namespaces/{namespace}/{plural}', 'GET', [e2e-predictor] path_params, [e2e-predictor] query_params, [e2e-predictor] header_params, [e2e-predictor] body=body_params, [e2e-predictor] post_params=form_params, [e2e-predictor] files=local_var_files, [e2e-predictor] response_type='object', # noqa: E501 [e2e-predictor] auth_settings=auth_settings, [e2e-predictor] async_req=local_var_params.get('async_req'), [e2e-predictor] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-predictor] _preload_content=local_var_params.get('_preload_content', True), [e2e-predictor] _request_timeout=local_var_params.get('_request_timeout'), [e2e-predictor] collection_formats=collection_formats) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:2594: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}' [e2e-predictor] method = 'GET' [e2e-predictor] path_params = {'group': 'gateway.networking.k8s.io', 'namespace': 'kserve-ci-e2e-test', 'plural': 'gateways', 'version': 'v1'} [e2e-predictor] query_params = [] [e2e-predictor] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] body = None, post_params = [], files = {}, response_type = 'object' [e2e-predictor] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-predictor] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-predictor] _host = None [e2e-predictor] [e2e-predictor] def call_api(self, resource_path, method, [e2e-predictor] path_params=None, query_params=None, header_params=None, [e2e-predictor] body=None, post_params=None, files=None, [e2e-predictor] response_type=None, auth_settings=None, async_req=None, [e2e-predictor] _return_http_data_only=None, collection_formats=None, [e2e-predictor] _preload_content=True, _request_timeout=None, _host=None): [e2e-predictor] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-predictor] [e2e-predictor] To make an async_req request, set the async_req parameter. [e2e-predictor] [e2e-predictor] :param resource_path: Path to method endpoint. [e2e-predictor] :param method: Method to call. [e2e-predictor] :param path_params: Path parameters in the url. [e2e-predictor] :param query_params: Query parameters in the url. [e2e-predictor] :param header_params: Header parameters to be [e2e-predictor] placed in the request header. [e2e-predictor] :param body: Request body. [e2e-predictor] :param post_params dict: Request post form parameters, [e2e-predictor] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-predictor] :param auth_settings list: Auth Settings names for the request. [e2e-predictor] :param response: Response data type. [e2e-predictor] :param files dict: key -> filename, value -> filepath, [e2e-predictor] for `multipart/form-data`. [e2e-predictor] :param async_req bool: execute request asynchronously [e2e-predictor] :param _return_http_data_only: response data without head status code [e2e-predictor] and headers [e2e-predictor] :param collection_formats: dict of collection formats for path, query, [e2e-predictor] header, and post parameters. [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] :return: [e2e-predictor] If async_req parameter is True, [e2e-predictor] the request will be called asynchronously. [e2e-predictor] The method will return the request thread. [e2e-predictor] If parameter async_req is False or missing, [e2e-predictor] then the method will return the response directly. [e2e-predictor] """ [e2e-predictor] if not async_req: [e2e-predictor] > return self.__call_api(resource_path, method, [e2e-predictor] path_params, query_params, header_params, [e2e-predictor] body, post_params, files, [e2e-predictor] response_type, auth_settings, [e2e-predictor] _return_http_data_only, collection_formats, [e2e-predictor] _preload_content, _request_timeout, _host) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] resource_path = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] method = 'GET' [e2e-predictor] path_params = [('group', 'gateway.networking.k8s.io'), ('version', 'v1'), ('namespace', 'kserve-ci-e2e-test'), ('plural', 'gateways')] [e2e-predictor] query_params = [] [e2e-predictor] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] body = None, post_params = [], files = {}, response_type = 'object' [e2e-predictor] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-predictor] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-predictor] _host = None [e2e-predictor] [e2e-predictor] def __call_api( [e2e-predictor] self, resource_path, method, path_params=None, [e2e-predictor] query_params=None, header_params=None, body=None, post_params=None, [e2e-predictor] files=None, response_type=None, auth_settings=None, [e2e-predictor] _return_http_data_only=None, collection_formats=None, [e2e-predictor] _preload_content=True, _request_timeout=None, _host=None): [e2e-predictor] [e2e-predictor] config = self.configuration [e2e-predictor] [e2e-predictor] # header parameters [e2e-predictor] header_params = header_params or {} [e2e-predictor] header_params.update(self.default_headers) [e2e-predictor] if self.cookie: [e2e-predictor] header_params['Cookie'] = self.cookie [e2e-predictor] if header_params: [e2e-predictor] header_params = self.sanitize_for_serialization(header_params) [e2e-predictor] header_params = dict(self.parameters_to_tuples(header_params, [e2e-predictor] collection_formats)) [e2e-predictor] [e2e-predictor] # path parameters [e2e-predictor] if path_params: [e2e-predictor] path_params = self.sanitize_for_serialization(path_params) [e2e-predictor] path_params = self.parameters_to_tuples(path_params, [e2e-predictor] collection_formats) [e2e-predictor] for k, v in path_params: [e2e-predictor] # specified safe chars, encode everything [e2e-predictor] resource_path = resource_path.replace( [e2e-predictor] '{%s}' % k, [e2e-predictor] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # query parameters [e2e-predictor] if query_params: [e2e-predictor] query_params = self.sanitize_for_serialization(query_params) [e2e-predictor] query_params = self.parameters_to_tuples(query_params, [e2e-predictor] collection_formats) [e2e-predictor] [e2e-predictor] # post parameters [e2e-predictor] if post_params or files: [e2e-predictor] post_params = post_params if post_params else [] [e2e-predictor] post_params = self.sanitize_for_serialization(post_params) [e2e-predictor] post_params = self.parameters_to_tuples(post_params, [e2e-predictor] collection_formats) [e2e-predictor] post_params.extend(self.files_parameters(files)) [e2e-predictor] [e2e-predictor] # auth setting [e2e-predictor] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-predictor] [e2e-predictor] # body [e2e-predictor] if body: [e2e-predictor] body = self.sanitize_for_serialization(body) [e2e-predictor] [e2e-predictor] # request url [e2e-predictor] if _host is None: [e2e-predictor] url = self.configuration.host + resource_path [e2e-predictor] else: [e2e-predictor] # use server/host defined in path or operation instead [e2e-predictor] url = _host + resource_path [e2e-predictor] [e2e-predictor] # perform request and return response [e2e-predictor] > response_data = self.request( [e2e-predictor] method, url, query_params=query_params, headers=header_params, [e2e-predictor] post_params=post_params, body=body, [e2e-predictor] _preload_content=_preload_content, [e2e-predictor] _request_timeout=_request_timeout) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] query_params = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] post_params = [], body = None, _preload_content = True, _request_timeout = None [e2e-predictor] [e2e-predictor] def request(self, method, url, query_params=None, headers=None, [e2e-predictor] post_params=None, body=None, _preload_content=True, [e2e-predictor] _request_timeout=None): [e2e-predictor] """Makes the HTTP request using RESTClient.""" [e2e-predictor] if method == "GET": [e2e-predictor] > return self.rest_client.GET(url, [e2e-predictor] query_params=query_params, [e2e-predictor] _preload_content=_preload_content, [e2e-predictor] _request_timeout=_request_timeout, [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:373: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] query_params = [], _preload_content = True, _request_timeout = None [e2e-predictor] [e2e-predictor] def GET(self, url, headers=None, query_params=None, _preload_content=True, [e2e-predictor] _request_timeout=None): [e2e-predictor] > return self.request("GET", url, [e2e-predictor] headers=headers, [e2e-predictor] _preload_content=_preload_content, [e2e-predictor] _request_timeout=_request_timeout, [e2e-predictor] query_params=query_params) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:244: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] query_params = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] body = None, post_params = {}, _preload_content = True, _request_timeout = None [e2e-predictor] [e2e-predictor] def request(self, method, url, query_params=None, headers=None, [e2e-predictor] body=None, post_params=None, _preload_content=True, [e2e-predictor] _request_timeout=None): [e2e-predictor] """Perform requests. [e2e-predictor] [e2e-predictor] :param method: http request method [e2e-predictor] :param url: http request url [e2e-predictor] :param query_params: query parameters in the url [e2e-predictor] :param headers: http request headers [e2e-predictor] :param body: request json body, for `application/json` [e2e-predictor] :param post_params: request post parameters, [e2e-predictor] `application/x-www-form-urlencoded` [e2e-predictor] and `multipart/form-data` [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] """ [e2e-predictor] method = method.upper() [e2e-predictor] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-predictor] 'PATCH', 'OPTIONS'] [e2e-predictor] [e2e-predictor] if post_params and body: [e2e-predictor] raise ApiValueError( [e2e-predictor] "body parameter cannot be used with post_params parameter." [e2e-predictor] ) [e2e-predictor] [e2e-predictor] post_params = post_params or {} [e2e-predictor] headers = headers or {} [e2e-predictor] [e2e-predictor] timeout = None [e2e-predictor] if _request_timeout: [e2e-predictor] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-predictor] timeout = urllib3.Timeout(total=_request_timeout) [e2e-predictor] elif (isinstance(_request_timeout, tuple) and [e2e-predictor] len(_request_timeout) == 2): [e2e-predictor] timeout = urllib3.Timeout( [e2e-predictor] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-predictor] [e2e-predictor] if 'Content-Type' not in headers: [e2e-predictor] headers['Content-Type'] = 'application/json' [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-predictor] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-predictor] if query_params: [e2e-predictor] url += '?' + urlencode(query_params) [e2e-predictor] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-predictor] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-predictor] if headers['Content-Type'] == 'application/json-patch+json': [e2e-predictor] if not isinstance(body, list): [e2e-predictor] headers['Content-Type'] = \ [e2e-predictor] 'application/strategic-merge-patch+json' [e2e-predictor] request_body = None [e2e-predictor] if body is not None: [e2e-predictor] request_body = json.dumps(body) [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] body=request_body, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] fields=post_params, [e2e-predictor] encode_multipart=False, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] elif headers['Content-Type'] == 'multipart/form-data': [e2e-predictor] # must del headers['Content-Type'], or the correct [e2e-predictor] # Content-Type which generated by urllib3 will be [e2e-predictor] # overwritten. [e2e-predictor] del headers['Content-Type'] [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] fields=post_params, [e2e-predictor] encode_multipart=True, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] # Pass a `string` parameter directly in the body to support [e2e-predictor] # other content types than Json when `body` argument is [e2e-predictor] # provided in serialized form [e2e-predictor] elif isinstance(body, str) or isinstance(body, bytes): [e2e-predictor] request_body = body [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] body=request_body, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] else: [e2e-predictor] # Cannot generate the request from given parameters [e2e-predictor] msg = """Cannot prepare a request message for provided [e2e-predictor] arguments. Please check that your arguments match [e2e-predictor] declared content type.""" [e2e-predictor] raise ApiException(status=0, reason=msg) [e2e-predictor] # For `GET`, `HEAD` [e2e-predictor] else: [e2e-predictor] > r = self.pool_manager.request(method, url, [e2e-predictor] fields=query_params, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:217: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None, fields = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] json = None, urlopen_kw = {'preload_content': True, 'timeout': None} [e2e-predictor] [e2e-predictor] def request( [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] fields: _TYPE_FIELDS | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] json: typing.Any | None = None, [e2e-predictor] **urlopen_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Make a request using :meth:`urlopen` with the appropriate encoding of [e2e-predictor] ``fields`` based on the ``method`` used. [e2e-predictor] [e2e-predictor] This is a convenience method that requires the least amount of manual [e2e-predictor] effort. It can be used in most situations, while still having the [e2e-predictor] option to drop down to more specific methods when necessary, such as [e2e-predictor] :meth:`request_encode_url`, :meth:`request_encode_body`, [e2e-predictor] or even the lowest level :meth:`urlopen`. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param fields: [e2e-predictor] Data to encode and send in the URL or request body, depending on ``method``. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param json: [e2e-predictor] Data to encode and send as JSON with UTF-encoded in the request body. [e2e-predictor] The ``"Content-Type"`` header will be set to ``"application/json"`` [e2e-predictor] unless specified otherwise. [e2e-predictor] """ [e2e-predictor] method = method.upper() [e2e-predictor] [e2e-predictor] if json is not None and body is not None: [e2e-predictor] raise TypeError( [e2e-predictor] "request got values for both 'body' and 'json' parameters which are mutually exclusive" [e2e-predictor] ) [e2e-predictor] [e2e-predictor] if json is not None: [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not ("content-type" in map(str.lower, headers.keys())): [e2e-predictor] headers = HTTPHeaderDict(headers) [e2e-predictor] headers["Content-Type"] = "application/json" [e2e-predictor] [e2e-predictor] body = _json.dumps(json, separators=(",", ":"), ensure_ascii=False).encode( [e2e-predictor] "utf-8" [e2e-predictor] ) [e2e-predictor] [e2e-predictor] if body is not None: [e2e-predictor] urlopen_kw["body"] = body [e2e-predictor] [e2e-predictor] if method in self._encode_url_methods: [e2e-predictor] > return self.request_encode_url( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] fields=fields, # type: ignore[arg-type] [e2e-predictor] headers=headers, [e2e-predictor] **urlopen_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/_request_methods.py:135: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] fields = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] urlopen_kw = {'preload_content': True, 'timeout': None} [e2e-predictor] extra_kw = {'headers': {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'}, 'preload_content': True, 'timeout': None} [e2e-predictor] [e2e-predictor] def request_encode_url( [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] fields: _TYPE_ENCODE_URL_FIELDS | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] **urlopen_kw: str, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Make a request using :meth:`urlopen` with the ``fields`` encoded in [e2e-predictor] the url. This is useful for request methods like GET, HEAD, DELETE, etc. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param fields: [e2e-predictor] Data to encode and send in the URL. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] """ [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] extra_kw: dict[str, typing.Any] = {"headers": headers} [e2e-predictor] extra_kw.update(urlopen_kw) [e2e-predictor] [e2e-predictor] if fields: [e2e-predictor] url += "?" + urlencode(fields) [e2e-predictor] [e2e-predictor] > return self.urlopen(method, url, **extra_kw) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/_request_methods.py:182: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] redirect = True [e2e-predictor] kw = {'assert_same_host': False, 'headers': {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'}, 'preload_content': True, 'redirect': False, ...} [e2e-predictor] u = Url(scheme='https', auth=None, host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443, path='/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways', query=None, fragment=None) [e2e-predictor] conn = [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, method: str, url: str, redirect: bool = True, **kw: typing.Any [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Same as :meth:`urllib3.HTTPConnectionPool.urlopen` [e2e-predictor] with custom cross-host redirect logic and only sends the request-uri [e2e-predictor] portion of the ``url``. [e2e-predictor] [e2e-predictor] The given ``url`` parameter must be absolute, such that an appropriate [e2e-predictor] :class:`urllib3.connectionpool.ConnectionPool` can be chosen for it. [e2e-predictor] """ [e2e-predictor] u = parse_url(url) [e2e-predictor] [e2e-predictor] if u.scheme is None: [e2e-predictor] warnings.warn( [e2e-predictor] "URLs without a scheme (ie 'https://') are deprecated and will raise an error " [e2e-predictor] "in urllib3 v3.0. To avoid this FutureWarning ensure all URLs " [e2e-predictor] "start with 'https://' or 'http://'. Read more in this issue: " [e2e-predictor] "https://github.com/urllib3/urllib3/issues/2920", [e2e-predictor] category=FutureWarning, [e2e-predictor] stacklevel=2, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme) [e2e-predictor] [e2e-predictor] kw["assert_same_host"] = False [e2e-predictor] kw["redirect"] = False [e2e-predictor] [e2e-predictor] if "headers" not in kw: [e2e-predictor] kw["headers"] = self.headers [e2e-predictor] [e2e-predictor] if self._proxy_requires_url_absolute_form(u): [e2e-predictor] response = conn.urlopen(method, url, **kw) [e2e-predictor] else: [e2e-predictor] > response = conn.urlopen(method, u.request_uri, **kw) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/poolmanager.py:457: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=2, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False [e2e-predictor] err = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] retries.sleep() [e2e-predictor] [e2e-predictor] # Keep track of the error for the retry warning. [e2e-predictor] err = e [e2e-predictor] [e2e-predictor] finally: [e2e-predictor] if not clean_exit: [e2e-predictor] # We hit some kind of exception, handled or otherwise. We need [e2e-predictor] # to throw the connection away unless explicitly told not to. [e2e-predictor] # Close the connection, set the variable to None, and make sure [e2e-predictor] # we put the None back in the pool to avoid leaking it. [e2e-predictor] if conn: [e2e-predictor] conn.close() [e2e-predictor] conn = None [e2e-predictor] release_this_conn = True [e2e-predictor] [e2e-predictor] if release_this_conn: [e2e-predictor] # Put the connection back to be reused. If the connection is [e2e-predictor] # expired then it will be None, which will get replaced with a [e2e-predictor] # fresh connection during _get_conn. [e2e-predictor] self._put_conn(conn) [e2e-predictor] [e2e-predictor] if not conn: [e2e-predictor] # Try again [e2e-predictor] log.warning( [e2e-predictor] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-predictor] ) [e2e-predictor] > return self.urlopen( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] body, [e2e-predictor] headers, [e2e-predictor] retries, [e2e-predictor] redirect, [e2e-predictor] assert_same_host, [e2e-predictor] timeout=timeout, [e2e-predictor] pool_timeout=pool_timeout, [e2e-predictor] release_conn=release_conn, [e2e-predictor] chunked=chunked, [e2e-predictor] body_pos=body_pos, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:872: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=1, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False [e2e-predictor] err = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] retries.sleep() [e2e-predictor] [e2e-predictor] # Keep track of the error for the retry warning. [e2e-predictor] err = e [e2e-predictor] [e2e-predictor] finally: [e2e-predictor] if not clean_exit: [e2e-predictor] # We hit some kind of exception, handled or otherwise. We need [e2e-predictor] # to throw the connection away unless explicitly told not to. [e2e-predictor] # Close the connection, set the variable to None, and make sure [e2e-predictor] # we put the None back in the pool to avoid leaking it. [e2e-predictor] if conn: [e2e-predictor] conn.close() [e2e-predictor] conn = None [e2e-predictor] release_this_conn = True [e2e-predictor] [e2e-predictor] if release_this_conn: [e2e-predictor] # Put the connection back to be reused. If the connection is [e2e-predictor] # expired then it will be None, which will get replaced with a [e2e-predictor] # fresh connection during _get_conn. [e2e-predictor] self._put_conn(conn) [e2e-predictor] [e2e-predictor] if not conn: [e2e-predictor] # Try again [e2e-predictor] log.warning( [e2e-predictor] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-predictor] ) [e2e-predictor] > return self.urlopen( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] body, [e2e-predictor] headers, [e2e-predictor] retries, [e2e-predictor] redirect, [e2e-predictor] assert_same_host, [e2e-predictor] timeout=timeout, [e2e-predictor] pool_timeout=pool_timeout, [e2e-predictor] release_conn=release_conn, [e2e-predictor] chunked=chunked, [e2e-predictor] body_pos=body_pos, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:872: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False [e2e-predictor] err = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] retries.sleep() [e2e-predictor] [e2e-predictor] # Keep track of the error for the retry warning. [e2e-predictor] err = e [e2e-predictor] [e2e-predictor] finally: [e2e-predictor] if not clean_exit: [e2e-predictor] # We hit some kind of exception, handled or otherwise. We need [e2e-predictor] # to throw the connection away unless explicitly told not to. [e2e-predictor] # Close the connection, set the variable to None, and make sure [e2e-predictor] # we put the None back in the pool to avoid leaking it. [e2e-predictor] if conn: [e2e-predictor] conn.close() [e2e-predictor] conn = None [e2e-predictor] release_this_conn = True [e2e-predictor] [e2e-predictor] if release_this_conn: [e2e-predictor] # Put the connection back to be reused. If the connection is [e2e-predictor] # expired then it will be None, which will get replaced with a [e2e-predictor] # fresh connection during _get_conn. [e2e-predictor] self._put_conn(conn) [e2e-predictor] [e2e-predictor] if not conn: [e2e-predictor] # Try again [e2e-predictor] log.warning( [e2e-predictor] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-predictor] ) [e2e-predictor] > return self.urlopen( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] body, [e2e-predictor] headers, [e2e-predictor] retries, [e2e-predictor] redirect, [e2e-predictor] assert_same_host, [e2e-predictor] timeout=timeout, [e2e-predictor] pool_timeout=pool_timeout, [e2e-predictor] release_conn=release_conn, [e2e-predictor] chunked=chunked, [e2e-predictor] body_pos=body_pos, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:872: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False, err = None [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] > retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:842: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] response = None [e2e-predictor] error = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] _pool = [e2e-predictor] _stacktrace = [e2e-predictor] [e2e-predictor] def increment( [e2e-predictor] self, [e2e-predictor] method: str | None = None, [e2e-predictor] url: str | None = None, [e2e-predictor] response: BaseHTTPResponse | None = None, [e2e-predictor] error: Exception | None = None, [e2e-predictor] _pool: ConnectionPool | None = None, [e2e-predictor] _stacktrace: TracebackType | None = None, [e2e-predictor] ) -> Self: [e2e-predictor] """Return a new Retry object with incremented retry counters. [e2e-predictor] [e2e-predictor] :param response: A response object, or None, if the server did not [e2e-predictor] return a response. [e2e-predictor] :type response: :class:`~urllib3.response.BaseHTTPResponse` [e2e-predictor] :param Exception error: An error encountered during the request, or [e2e-predictor] None if the response was received successfully. [e2e-predictor] [e2e-predictor] :return: A new ``Retry`` object. [e2e-predictor] """ [e2e-predictor] if self.total is False and error: [e2e-predictor] # Disabled, indicate to re-raise the error. [e2e-predictor] raise reraise(type(error), error, _stacktrace) [e2e-predictor] [e2e-predictor] total = self.total [e2e-predictor] if total is not None: [e2e-predictor] total -= 1 [e2e-predictor] [e2e-predictor] connect = self.connect [e2e-predictor] read = self.read [e2e-predictor] redirect = self.redirect [e2e-predictor] status_count = self.status [e2e-predictor] other = self.other [e2e-predictor] cause = "unknown" [e2e-predictor] status = None [e2e-predictor] redirect_location = None [e2e-predictor] [e2e-predictor] if error and self._is_connection_error(error): [e2e-predictor] # Connect retry? [e2e-predictor] if connect is False: [e2e-predictor] raise reraise(type(error), error, _stacktrace) [e2e-predictor] elif connect is not None: [e2e-predictor] connect -= 1 [e2e-predictor] [e2e-predictor] elif error and self._is_read_error(error): [e2e-predictor] # Read retry? [e2e-predictor] if read is False or method is None or not self._is_method_retryable(method): [e2e-predictor] raise reraise(type(error), error, _stacktrace) [e2e-predictor] elif read is not None: [e2e-predictor] read -= 1 [e2e-predictor] [e2e-predictor] elif error: [e2e-predictor] # Other retry? [e2e-predictor] if other is not None: [e2e-predictor] other -= 1 [e2e-predictor] [e2e-predictor] elif response and response.get_redirect_location(): [e2e-predictor] # Redirect retry? [e2e-predictor] if redirect is not None: [e2e-predictor] redirect -= 1 [e2e-predictor] cause = "too many redirects" [e2e-predictor] response_redirect_location = response.get_redirect_location() [e2e-predictor] if response_redirect_location: [e2e-predictor] redirect_location = response_redirect_location [e2e-predictor] status = response.status [e2e-predictor] [e2e-predictor] else: [e2e-predictor] # Incrementing because of a server error like a 500 in [e2e-predictor] # status_forcelist and the given method is in the allowed_methods [e2e-predictor] cause = ResponseError.GENERIC_ERROR [e2e-predictor] if response and response.status: [e2e-predictor] if status_count is not None: [e2e-predictor] status_count -= 1 [e2e-predictor] cause = ResponseError.SPECIFIC_ERROR.format(status_code=response.status) [e2e-predictor] status = response.status [e2e-predictor] [e2e-predictor] history = self.history + ( [e2e-predictor] RequestHistory(method, url, error, status, redirect_location), [e2e-predictor] ) [e2e-predictor] [e2e-predictor] new_retry = self.new( [e2e-predictor] total=total, [e2e-predictor] connect=connect, [e2e-predictor] read=read, [e2e-predictor] redirect=redirect, [e2e-predictor] status=status_count, [e2e-predictor] other=other, [e2e-predictor] history=history, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] if new_retry.is_exhausted(): [e2e-predictor] reason = error or ResponseError(cause) [e2e-predictor] > raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] [e2e-predictor] E urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Max retries exceeded with url: /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways (Caused by NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/retry.py:543: MaxRetryError [e2e-predictor] ------------------------------ Captured log setup ------------------------------ [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways [e2e-predictor] ___________________ ERROR at setup of test_tensorflow_kserve ___________________ [e2e-predictor] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-predictor] [e2e-predictor] self = [e2e-predictor] [e2e-predictor] def _new_conn(self) -> socket.socket: [e2e-predictor] """Establish a socket connection and set nodelay settings on it. [e2e-predictor] [e2e-predictor] :return: New socket connection. [e2e-predictor] """ [e2e-predictor] try: [e2e-predictor] > sock = connection.create_connection( [e2e-predictor] (self._dns_host, self.port), [e2e-predictor] self.timeout, [e2e-predictor] source_address=self.source_address, [e2e-predictor] socket_options=self.socket_options, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:204: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] address = ('a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', 6443) [e2e-predictor] timeout = None, source_address = None, socket_options = [(6, 1, 1)] [e2e-predictor] [e2e-predictor] def create_connection( [e2e-predictor] address: tuple[str, int], [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] source_address: tuple[str, int] | None = None, [e2e-predictor] socket_options: _TYPE_SOCKET_OPTIONS | None = None, [e2e-predictor] ) -> socket.socket: [e2e-predictor] """Connect to *address* and return the socket object. [e2e-predictor] [e2e-predictor] Convenience function. Connect to *address* (a 2-tuple ``(host, [e2e-predictor] port)``) and return the socket object. Passing the optional [e2e-predictor] *timeout* parameter will set the timeout on the socket instance [e2e-predictor] before attempting to connect. If no *timeout* is supplied, the [e2e-predictor] global default timeout setting returned by :func:`socket.getdefaulttimeout` [e2e-predictor] is used. If *source_address* is set it must be a tuple of (host, port) [e2e-predictor] for the socket to bind as a source address before making the connection. [e2e-predictor] An host of '' or port 0 tells the OS to use the default. [e2e-predictor] """ [e2e-predictor] [e2e-predictor] host, port = address [e2e-predictor] if host.startswith("["): [e2e-predictor] host = host.strip("[]") [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Using the value from allowed_gai_family() in the context of getaddrinfo lets [e2e-predictor] # us select whether to work with IPv4 DNS records, IPv6 records, or both. [e2e-predictor] # The original create_connection function always returns all records. [e2e-predictor] family = allowed_gai_family() [e2e-predictor] [e2e-predictor] try: [e2e-predictor] host.encode("idna") [e2e-predictor] except UnicodeError: [e2e-predictor] raise LocationParseError(f"'{host}', label empty or too long") from None [e2e-predictor] [e2e-predictor] > for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/connection.py:60: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] host = 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' [e2e-predictor] port = 6443, family = [e2e-predictor] type = , proto = 0, flags = 0 [e2e-predictor] [e2e-predictor] def getaddrinfo(host, port, family=0, type=0, proto=0, flags=0): [e2e-predictor] """Resolve host and port into list of address info entries. [e2e-predictor] [e2e-predictor] Translate the host/port argument into a sequence of 5-tuples that contain [e2e-predictor] all the necessary arguments for creating a socket connected to that service. [e2e-predictor] host is a domain name, a string representation of an IPv4/v6 address or [e2e-predictor] None. port is a string service name such as 'http', a numeric port number or [e2e-predictor] None. By passing None as the value of host and port, you can pass NULL to [e2e-predictor] the underlying C API. [e2e-predictor] [e2e-predictor] The family, type and proto arguments can be optionally specified in order to [e2e-predictor] narrow the list of addresses returned. Passing zero as a value for each of [e2e-predictor] these arguments selects the full range of results. [e2e-predictor] """ [e2e-predictor] # We override this function since we want to translate the numeric family [e2e-predictor] # and socket type values to enum constants. [e2e-predictor] addrlist = [] [e2e-predictor] > for res in _socket.getaddrinfo(host, port, family, type, proto, flags): [e2e-predictor] E socket.gaierror: [Errno -2] Name or service not known [e2e-predictor] [e2e-predictor] /usr/lib64/python3.11/socket.py:974: gaierror [e2e-predictor] [e2e-predictor] The above exception was the direct cause of the following exception: [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False, err = None [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] > response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:788: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] conn = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] timeout = Timeout(connect=None, read=None, total=None), chunked = False [e2e-predictor] response_conn = None, preload_content = True, decode_content = True [e2e-predictor] enforce_content_length = True [e2e-predictor] [e2e-predictor] def _make_request( [e2e-predictor] self, [e2e-predictor] conn: BaseHTTPConnection, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | None = None, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] chunked: bool = False, [e2e-predictor] response_conn: BaseHTTPConnection | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] enforce_content_length: bool = True, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Perform a request on a given urllib connection object taken from our [e2e-predictor] pool. [e2e-predictor] [e2e-predictor] :param conn: [e2e-predictor] a connection from one of our connection pools [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] Pass ``None`` to retry until you receive a response. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param response_conn: [e2e-predictor] Set this to ``None`` if you will handle releasing the connection or [e2e-predictor] set the connection to have the response release it. [e2e-predictor] [e2e-predictor] :param preload_content: [e2e-predictor] If True, the response's body will be preloaded during construction. [e2e-predictor] [e2e-predictor] :param decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param enforce_content_length: [e2e-predictor] Enforce content length checking. Body returned by server must match [e2e-predictor] value of Content-Length header, if present. Otherwise, raise error. [e2e-predictor] """ [e2e-predictor] self.num_requests += 1 [e2e-predictor] [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] timeout_obj.start_connect() [e2e-predictor] conn.timeout = Timeout.resolve_default_timeout(timeout_obj.connect_timeout) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Trigger any extra validation we need to do. [e2e-predictor] try: [e2e-predictor] self._validate_conn(conn) [e2e-predictor] except (SocketTimeout, BaseSSLError) as e: [e2e-predictor] self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # _validate_conn() starts the connection to an HTTPS proxy [e2e-predictor] # so we need to wrap errors with 'ProxyError' here too. [e2e-predictor] except ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] BaseSSLError, [e2e-predictor] CertificateError, [e2e-predictor] SSLError, [e2e-predictor] ) as e: [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] # If the connection didn't successfully connect to it's proxy [e2e-predictor] # then there [e2e-predictor] if isinstance( [e2e-predictor] new_e, (OSError, NewConnectionError, TimeoutError, SSLError) [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] > raise new_e [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:488: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] conn = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] timeout = Timeout(connect=None, read=None, total=None), chunked = False [e2e-predictor] response_conn = None, preload_content = True, decode_content = True [e2e-predictor] enforce_content_length = True [e2e-predictor] [e2e-predictor] def _make_request( [e2e-predictor] self, [e2e-predictor] conn: BaseHTTPConnection, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | None = None, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] chunked: bool = False, [e2e-predictor] response_conn: BaseHTTPConnection | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] enforce_content_length: bool = True, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Perform a request on a given urllib connection object taken from our [e2e-predictor] pool. [e2e-predictor] [e2e-predictor] :param conn: [e2e-predictor] a connection from one of our connection pools [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] Pass ``None`` to retry until you receive a response. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param response_conn: [e2e-predictor] Set this to ``None`` if you will handle releasing the connection or [e2e-predictor] set the connection to have the response release it. [e2e-predictor] [e2e-predictor] :param preload_content: [e2e-predictor] If True, the response's body will be preloaded during construction. [e2e-predictor] [e2e-predictor] :param decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param enforce_content_length: [e2e-predictor] Enforce content length checking. Body returned by server must match [e2e-predictor] value of Content-Length header, if present. Otherwise, raise error. [e2e-predictor] """ [e2e-predictor] self.num_requests += 1 [e2e-predictor] [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] timeout_obj.start_connect() [e2e-predictor] conn.timeout = Timeout.resolve_default_timeout(timeout_obj.connect_timeout) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Trigger any extra validation we need to do. [e2e-predictor] try: [e2e-predictor] > self._validate_conn(conn) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:464: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] conn = [e2e-predictor] [e2e-predictor] def _validate_conn(self, conn: BaseHTTPConnection) -> None: [e2e-predictor] """ [e2e-predictor] Called right before a request is made, after the socket is created. [e2e-predictor] """ [e2e-predictor] super()._validate_conn(conn) [e2e-predictor] [e2e-predictor] # Force connect early to allow us to validate the connection. [e2e-predictor] if conn.is_closed: [e2e-predictor] > conn.connect() [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:1106: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] [e2e-predictor] def connect(self) -> None: [e2e-predictor] # Today we don't need to be doing this step before the /actual/ socket [e2e-predictor] # connection, however in the future we'll need to decide whether to [e2e-predictor] # create a new socket or re-use an existing "shared" socket as a part [e2e-predictor] # of the HTTP/2 handshake dance. [e2e-predictor] if self._tunnel_host is not None and self._tunnel_port is not None: [e2e-predictor] probe_http2_host = self._tunnel_host [e2e-predictor] probe_http2_port = self._tunnel_port [e2e-predictor] else: [e2e-predictor] probe_http2_host = self.host [e2e-predictor] probe_http2_port = self.port [e2e-predictor] [e2e-predictor] # Check if the target origin supports HTTP/2. [e2e-predictor] # If the value comes back as 'None' it means that the current thread [e2e-predictor] # is probing for HTTP/2 support. Otherwise, we're waiting for another [e2e-predictor] # probe to complete, or we get a value right away. [e2e-predictor] target_supports_http2: bool | None [e2e-predictor] if "h2" in ssl_.ALPN_PROTOCOLS: [e2e-predictor] target_supports_http2 = http2_probe.acquire_and_get( [e2e-predictor] host=probe_http2_host, port=probe_http2_port [e2e-predictor] ) [e2e-predictor] else: [e2e-predictor] # If HTTP/2 isn't going to be offered it doesn't matter if [e2e-predictor] # the target supports HTTP/2. Don't want to make a probe. [e2e-predictor] target_supports_http2 = False [e2e-predictor] [e2e-predictor] if self._connect_callback is not None: [e2e-predictor] self._connect_callback( [e2e-predictor] "before connect", [e2e-predictor] thread_id=threading.get_ident(), [e2e-predictor] target_supports_http2=target_supports_http2, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] sock: socket.socket | ssl.SSLSocket [e2e-predictor] > self.sock = sock = self._new_conn() [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:759: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] [e2e-predictor] def _new_conn(self) -> socket.socket: [e2e-predictor] """Establish a socket connection and set nodelay settings on it. [e2e-predictor] [e2e-predictor] :return: New socket connection. [e2e-predictor] """ [e2e-predictor] try: [e2e-predictor] sock = connection.create_connection( [e2e-predictor] (self._dns_host, self.port), [e2e-predictor] self.timeout, [e2e-predictor] source_address=self.source_address, [e2e-predictor] socket_options=self.socket_options, [e2e-predictor] ) [e2e-predictor] except socket.gaierror as e: [e2e-predictor] > raise NameResolutionError(self.host, self, e) from e [e2e-predictor] E urllib3.exceptions.NameResolutionError: HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:211: NameResolutionError [e2e-predictor] [e2e-predictor] The above exception was the direct cause of the following exception: [e2e-predictor] [e2e-predictor] request = > [e2e-predictor] [e2e-predictor] @pytest.fixture(autouse=True) [e2e-predictor] def ensure_gateway_proxy_memory(request): [e2e-predictor] """After test setup creates gateways, patch them for proxy memory.""" [e2e-predictor] if not GATEWAY_PROXY_MEMORY: [e2e-predictor] return [e2e-predictor] [e2e-predictor] # Let test_case (llmisvc) create gateways first [e2e-predictor] [e2e-predictor] if "test_case" in request.fixturenames: [e2e-predictor] request.getfixturevalue("test_case") [e2e-predictor] [e2e-predictor] from kserve import KServeClient [e2e-predictor] [e2e-predictor] kserve_client = KServeClient( [e2e-predictor] config_file=os.environ.get("KUBECONFIG", "~/.kube/config") [e2e-predictor] ) [e2e-predictor] api_client = kserve_client.api_instance.api_client [e2e-predictor] [e2e-predictor] namespace = os.environ.get("KSERVE_TEST_NAMESPACE", "kserve-ci-e2e-test") [e2e-predictor] _ensure_configmap(namespace, api_client) [e2e-predictor] > patched = _patch_gateways(namespace, api_client) [e2e-predictor] [e2e-predictor] common/gateway_proxy_istio.py:194: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] namespace = 'kserve-ci-e2e-test' [e2e-predictor] api_client = [e2e-predictor] [e2e-predictor] def _patch_gateways(namespace, api_client): [e2e-predictor] """Patch all Gateways in namespace to include parametersRef. [e2e-predictor] [e2e-predictor] Returns the list of gateway names that were actually patched. [e2e-predictor] """ [e2e-predictor] custom_api = client.CustomObjectsApi(api_client) [e2e-predictor] > gateways = custom_api.list_namespaced_custom_object( [e2e-predictor] "gateway.networking.k8s.io", "v1", namespace, "gateways" [e2e-predictor] ) [e2e-predictor] [e2e-predictor] common/gateway_proxy_istio.py:117: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] group = 'gateway.networking.k8s.io', version = 'v1' [e2e-predictor] namespace = 'kserve-ci-e2e-test', plural = 'gateways' [e2e-predictor] kwargs = {'_return_http_data_only': True} [e2e-predictor] [e2e-predictor] def list_namespaced_custom_object(self, group, version, namespace, plural, **kwargs): # noqa: E501 [e2e-predictor] """list_namespaced_custom_object # noqa: E501 [e2e-predictor] [e2e-predictor] list or watch namespace scoped custom objects # noqa: E501 [e2e-predictor] This method makes a synchronous HTTP request by default. To make an [e2e-predictor] asynchronous HTTP request, please pass async_req=True [e2e-predictor] >>> thread = api.list_namespaced_custom_object(group, version, namespace, plural, async_req=True) [e2e-predictor] >>> result = thread.get() [e2e-predictor] [e2e-predictor] :param async_req bool: execute request asynchronously [e2e-predictor] :param str group: The custom resource's group name (required) [e2e-predictor] :param str version: The custom resource's version (required) [e2e-predictor] :param str namespace: The custom resource's namespace (required) [e2e-predictor] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-predictor] :param str pretty: If 'true', then the output is pretty printed. [e2e-predictor] :param bool allow_watch_bookmarks: allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored. [e2e-predictor] :param str _continue: The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\". This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications. [e2e-predictor] :param str field_selector: A selector to restrict the list of returned objects by their fields. Defaults to everything. [e2e-predictor] :param str label_selector: A selector to restrict the list of returned objects by their labels. Defaults to everything. [e2e-predictor] :param int limit: limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true. The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned. [e2e-predictor] :param str resource_version: When specified with a watch call, shows changes that occur after that particular version of a resource. Defaults to changes from the beginning of history. When specified for list: - if unset, then the result is returned from remote storage based on quorum-read flag; - if it's 0, then we simply return what we currently have in cache, no guarantee; - if set to non zero, then the result is at least as fresh as given rv. [e2e-predictor] :param str resource_version_match: resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details. Defaults to unset [e2e-predictor] :param int timeout_seconds: Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity. [e2e-predictor] :param bool watch: Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] :return: object [e2e-predictor] If the method is called asynchronously, [e2e-predictor] returns the request thread. [e2e-predictor] """ [e2e-predictor] kwargs['_return_http_data_only'] = True [e2e-predictor] > return self.list_namespaced_custom_object_with_http_info(group, version, namespace, plural, **kwargs) # noqa: E501 [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:2455: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] group = 'gateway.networking.k8s.io', version = 'v1' [e2e-predictor] namespace = 'kserve-ci-e2e-test', plural = 'gateways' [e2e-predictor] kwargs = {'_return_http_data_only': True} [e2e-predictor] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'pretty', 'allow_watch_bookmarks', ...], 'auth_settings': ['BearerToken'], 'body_params': None, ...} [e2e-predictor] all_params = ['group', 'version', 'namespace', 'plural', 'pretty', 'allow_watch_bookmarks', ...] [e2e-predictor] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-predictor] path_params = {'group': 'gateway.networking.k8s.io', 'namespace': 'kserve-ci-e2e-test', 'plural': 'gateways', 'version': 'v1'} [e2e-predictor] query_params = [] [e2e-predictor] [e2e-predictor] def list_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, **kwargs): # noqa: E501 [e2e-predictor] """list_namespaced_custom_object # noqa: E501 [e2e-predictor] [e2e-predictor] list or watch namespace scoped custom objects # noqa: E501 [e2e-predictor] This method makes a synchronous HTTP request by default. To make an [e2e-predictor] asynchronous HTTP request, please pass async_req=True [e2e-predictor] >>> thread = api.list_namespaced_custom_object_with_http_info(group, version, namespace, plural, async_req=True) [e2e-predictor] >>> result = thread.get() [e2e-predictor] [e2e-predictor] :param async_req bool: execute request asynchronously [e2e-predictor] :param str group: The custom resource's group name (required) [e2e-predictor] :param str version: The custom resource's version (required) [e2e-predictor] :param str namespace: The custom resource's namespace (required) [e2e-predictor] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-predictor] :param str pretty: If 'true', then the output is pretty printed. [e2e-predictor] :param bool allow_watch_bookmarks: allowWatchBookmarks requests watch events with type \"BOOKMARK\". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored. [e2e-predictor] :param str _continue: The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the \"next key\". This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications. [e2e-predictor] :param str field_selector: A selector to restrict the list of returned objects by their fields. Defaults to everything. [e2e-predictor] :param str label_selector: A selector to restrict the list of returned objects by their labels. Defaults to everything. [e2e-predictor] :param int limit: limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true. The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned. [e2e-predictor] :param str resource_version: When specified with a watch call, shows changes that occur after that particular version of a resource. Defaults to changes from the beginning of history. When specified for list: - if unset, then the result is returned from remote storage based on quorum-read flag; - if it's 0, then we simply return what we currently have in cache, no guarantee; - if set to non zero, then the result is at least as fresh as given rv. [e2e-predictor] :param str resource_version_match: resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details. Defaults to unset [e2e-predictor] :param int timeout_seconds: Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity. [e2e-predictor] :param bool watch: Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. [e2e-predictor] :param _return_http_data_only: response data without head status code [e2e-predictor] and headers [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-predictor] If the method is called asynchronously, [e2e-predictor] returns the request thread. [e2e-predictor] """ [e2e-predictor] [e2e-predictor] local_var_params = locals() [e2e-predictor] [e2e-predictor] all_params = [ [e2e-predictor] 'group', [e2e-predictor] 'version', [e2e-predictor] 'namespace', [e2e-predictor] 'plural', [e2e-predictor] 'pretty', [e2e-predictor] 'allow_watch_bookmarks', [e2e-predictor] '_continue', [e2e-predictor] 'field_selector', [e2e-predictor] 'label_selector', [e2e-predictor] 'limit', [e2e-predictor] 'resource_version', [e2e-predictor] 'resource_version_match', [e2e-predictor] 'timeout_seconds', [e2e-predictor] 'watch' [e2e-predictor] ] [e2e-predictor] all_params.extend( [e2e-predictor] [ [e2e-predictor] 'async_req', [e2e-predictor] '_return_http_data_only', [e2e-predictor] '_preload_content', [e2e-predictor] '_request_timeout' [e2e-predictor] ] [e2e-predictor] ) [e2e-predictor] [e2e-predictor] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-predictor] if key not in all_params: [e2e-predictor] raise ApiTypeError( [e2e-predictor] "Got an unexpected keyword argument '%s'" [e2e-predictor] " to method list_namespaced_custom_object" % key [e2e-predictor] ) [e2e-predictor] local_var_params[key] = val [e2e-predictor] del local_var_params['kwargs'] [e2e-predictor] # verify the required parameter 'group' is set [e2e-predictor] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['group'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `group` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] # verify the required parameter 'version' is set [e2e-predictor] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['version'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `version` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] # verify the required parameter 'namespace' is set [e2e-predictor] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['namespace'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `namespace` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] # verify the required parameter 'plural' is set [e2e-predictor] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-predictor] local_var_params['plural'] is None): # noqa: E501 [e2e-predictor] raise ApiValueError("Missing the required parameter `plural` when calling `list_namespaced_custom_object`") # noqa: E501 [e2e-predictor] [e2e-predictor] collection_formats = {} [e2e-predictor] [e2e-predictor] path_params = {} [e2e-predictor] if 'group' in local_var_params: [e2e-predictor] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-predictor] if 'version' in local_var_params: [e2e-predictor] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-predictor] if 'namespace' in local_var_params: [e2e-predictor] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-predictor] if 'plural' in local_var_params: [e2e-predictor] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-predictor] [e2e-predictor] query_params = [] [e2e-predictor] if 'pretty' in local_var_params and local_var_params['pretty'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('pretty', local_var_params['pretty'])) # noqa: E501 [e2e-predictor] if 'allow_watch_bookmarks' in local_var_params and local_var_params['allow_watch_bookmarks'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('allowWatchBookmarks', local_var_params['allow_watch_bookmarks'])) # noqa: E501 [e2e-predictor] if '_continue' in local_var_params and local_var_params['_continue'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('continue', local_var_params['_continue'])) # noqa: E501 [e2e-predictor] if 'field_selector' in local_var_params and local_var_params['field_selector'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('fieldSelector', local_var_params['field_selector'])) # noqa: E501 [e2e-predictor] if 'label_selector' in local_var_params and local_var_params['label_selector'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('labelSelector', local_var_params['label_selector'])) # noqa: E501 [e2e-predictor] if 'limit' in local_var_params and local_var_params['limit'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('limit', local_var_params['limit'])) # noqa: E501 [e2e-predictor] if 'resource_version' in local_var_params and local_var_params['resource_version'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('resourceVersion', local_var_params['resource_version'])) # noqa: E501 [e2e-predictor] if 'resource_version_match' in local_var_params and local_var_params['resource_version_match'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('resourceVersionMatch', local_var_params['resource_version_match'])) # noqa: E501 [e2e-predictor] if 'timeout_seconds' in local_var_params and local_var_params['timeout_seconds'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('timeoutSeconds', local_var_params['timeout_seconds'])) # noqa: E501 [e2e-predictor] if 'watch' in local_var_params and local_var_params['watch'] is not None: # noqa: E501 [e2e-predictor] query_params.append(('watch', local_var_params['watch'])) # noqa: E501 [e2e-predictor] [e2e-predictor] header_params = {} [e2e-predictor] [e2e-predictor] form_params = [] [e2e-predictor] local_var_files = {} [e2e-predictor] [e2e-predictor] body_params = None [e2e-predictor] # HTTP header `Accept` [e2e-predictor] header_params['Accept'] = self.api_client.select_header_accept( [e2e-predictor] ['application/json', 'application/json;stream=watch']) # noqa: E501 [e2e-predictor] [e2e-predictor] # Authentication setting [e2e-predictor] auth_settings = ['BearerToken'] # noqa: E501 [e2e-predictor] [e2e-predictor] > return self.api_client.call_api( [e2e-predictor] '/apis/{group}/{version}/namespaces/{namespace}/{plural}', 'GET', [e2e-predictor] path_params, [e2e-predictor] query_params, [e2e-predictor] header_params, [e2e-predictor] body=body_params, [e2e-predictor] post_params=form_params, [e2e-predictor] files=local_var_files, [e2e-predictor] response_type='object', # noqa: E501 [e2e-predictor] auth_settings=auth_settings, [e2e-predictor] async_req=local_var_params.get('async_req'), [e2e-predictor] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-predictor] _preload_content=local_var_params.get('_preload_content', True), [e2e-predictor] _request_timeout=local_var_params.get('_request_timeout'), [e2e-predictor] collection_formats=collection_formats) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:2594: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}' [e2e-predictor] method = 'GET' [e2e-predictor] path_params = {'group': 'gateway.networking.k8s.io', 'namespace': 'kserve-ci-e2e-test', 'plural': 'gateways', 'version': 'v1'} [e2e-predictor] query_params = [] [e2e-predictor] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] body = None, post_params = [], files = {}, response_type = 'object' [e2e-predictor] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-predictor] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-predictor] _host = None [e2e-predictor] [e2e-predictor] def call_api(self, resource_path, method, [e2e-predictor] path_params=None, query_params=None, header_params=None, [e2e-predictor] body=None, post_params=None, files=None, [e2e-predictor] response_type=None, auth_settings=None, async_req=None, [e2e-predictor] _return_http_data_only=None, collection_formats=None, [e2e-predictor] _preload_content=True, _request_timeout=None, _host=None): [e2e-predictor] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-predictor] [e2e-predictor] To make an async_req request, set the async_req parameter. [e2e-predictor] [e2e-predictor] :param resource_path: Path to method endpoint. [e2e-predictor] :param method: Method to call. [e2e-predictor] :param path_params: Path parameters in the url. [e2e-predictor] :param query_params: Query parameters in the url. [e2e-predictor] :param header_params: Header parameters to be [e2e-predictor] placed in the request header. [e2e-predictor] :param body: Request body. [e2e-predictor] :param post_params dict: Request post form parameters, [e2e-predictor] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-predictor] :param auth_settings list: Auth Settings names for the request. [e2e-predictor] :param response: Response data type. [e2e-predictor] :param files dict: key -> filename, value -> filepath, [e2e-predictor] for `multipart/form-data`. [e2e-predictor] :param async_req bool: execute request asynchronously [e2e-predictor] :param _return_http_data_only: response data without head status code [e2e-predictor] and headers [e2e-predictor] :param collection_formats: dict of collection formats for path, query, [e2e-predictor] header, and post parameters. [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] :return: [e2e-predictor] If async_req parameter is True, [e2e-predictor] the request will be called asynchronously. [e2e-predictor] The method will return the request thread. [e2e-predictor] If parameter async_req is False or missing, [e2e-predictor] then the method will return the response directly. [e2e-predictor] """ [e2e-predictor] if not async_req: [e2e-predictor] > return self.__call_api(resource_path, method, [e2e-predictor] path_params, query_params, header_params, [e2e-predictor] body, post_params, files, [e2e-predictor] response_type, auth_settings, [e2e-predictor] _return_http_data_only, collection_formats, [e2e-predictor] _preload_content, _request_timeout, _host) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] resource_path = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] method = 'GET' [e2e-predictor] path_params = [('group', 'gateway.networking.k8s.io'), ('version', 'v1'), ('namespace', 'kserve-ci-e2e-test'), ('plural', 'gateways')] [e2e-predictor] query_params = [] [e2e-predictor] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] body = None, post_params = [], files = {}, response_type = 'object' [e2e-predictor] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-predictor] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-predictor] _host = None [e2e-predictor] [e2e-predictor] def __call_api( [e2e-predictor] self, resource_path, method, path_params=None, [e2e-predictor] query_params=None, header_params=None, body=None, post_params=None, [e2e-predictor] files=None, response_type=None, auth_settings=None, [e2e-predictor] _return_http_data_only=None, collection_formats=None, [e2e-predictor] _preload_content=True, _request_timeout=None, _host=None): [e2e-predictor] [e2e-predictor] config = self.configuration [e2e-predictor] [e2e-predictor] # header parameters [e2e-predictor] header_params = header_params or {} [e2e-predictor] header_params.update(self.default_headers) [e2e-predictor] if self.cookie: [e2e-predictor] header_params['Cookie'] = self.cookie [e2e-predictor] if header_params: [e2e-predictor] header_params = self.sanitize_for_serialization(header_params) [e2e-predictor] header_params = dict(self.parameters_to_tuples(header_params, [e2e-predictor] collection_formats)) [e2e-predictor] [e2e-predictor] # path parameters [e2e-predictor] if path_params: [e2e-predictor] path_params = self.sanitize_for_serialization(path_params) [e2e-predictor] path_params = self.parameters_to_tuples(path_params, [e2e-predictor] collection_formats) [e2e-predictor] for k, v in path_params: [e2e-predictor] # specified safe chars, encode everything [e2e-predictor] resource_path = resource_path.replace( [e2e-predictor] '{%s}' % k, [e2e-predictor] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # query parameters [e2e-predictor] if query_params: [e2e-predictor] query_params = self.sanitize_for_serialization(query_params) [e2e-predictor] query_params = self.parameters_to_tuples(query_params, [e2e-predictor] collection_formats) [e2e-predictor] [e2e-predictor] # post parameters [e2e-predictor] if post_params or files: [e2e-predictor] post_params = post_params if post_params else [] [e2e-predictor] post_params = self.sanitize_for_serialization(post_params) [e2e-predictor] post_params = self.parameters_to_tuples(post_params, [e2e-predictor] collection_formats) [e2e-predictor] post_params.extend(self.files_parameters(files)) [e2e-predictor] [e2e-predictor] # auth setting [e2e-predictor] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-predictor] [e2e-predictor] # body [e2e-predictor] if body: [e2e-predictor] body = self.sanitize_for_serialization(body) [e2e-predictor] [e2e-predictor] # request url [e2e-predictor] if _host is None: [e2e-predictor] url = self.configuration.host + resource_path [e2e-predictor] else: [e2e-predictor] # use server/host defined in path or operation instead [e2e-predictor] url = _host + resource_path [e2e-predictor] [e2e-predictor] # perform request and return response [e2e-predictor] > response_data = self.request( [e2e-predictor] method, url, query_params=query_params, headers=header_params, [e2e-predictor] post_params=post_params, body=body, [e2e-predictor] _preload_content=_preload_content, [e2e-predictor] _request_timeout=_request_timeout) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] query_params = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] post_params = [], body = None, _preload_content = True, _request_timeout = None [e2e-predictor] [e2e-predictor] def request(self, method, url, query_params=None, headers=None, [e2e-predictor] post_params=None, body=None, _preload_content=True, [e2e-predictor] _request_timeout=None): [e2e-predictor] """Makes the HTTP request using RESTClient.""" [e2e-predictor] if method == "GET": [e2e-predictor] > return self.rest_client.GET(url, [e2e-predictor] query_params=query_params, [e2e-predictor] _preload_content=_preload_content, [e2e-predictor] _request_timeout=_request_timeout, [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:373: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] query_params = [], _preload_content = True, _request_timeout = None [e2e-predictor] [e2e-predictor] def GET(self, url, headers=None, query_params=None, _preload_content=True, [e2e-predictor] _request_timeout=None): [e2e-predictor] > return self.request("GET", url, [e2e-predictor] headers=headers, [e2e-predictor] _preload_content=_preload_content, [e2e-predictor] _request_timeout=_request_timeout, [e2e-predictor] query_params=query_params) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:244: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] query_params = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] body = None, post_params = {}, _preload_content = True, _request_timeout = None [e2e-predictor] [e2e-predictor] def request(self, method, url, query_params=None, headers=None, [e2e-predictor] body=None, post_params=None, _preload_content=True, [e2e-predictor] _request_timeout=None): [e2e-predictor] """Perform requests. [e2e-predictor] [e2e-predictor] :param method: http request method [e2e-predictor] :param url: http request url [e2e-predictor] :param query_params: query parameters in the url [e2e-predictor] :param headers: http request headers [e2e-predictor] :param body: request json body, for `application/json` [e2e-predictor] :param post_params: request post parameters, [e2e-predictor] `application/x-www-form-urlencoded` [e2e-predictor] and `multipart/form-data` [e2e-predictor] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-predictor] be returned without reading/decoding response [e2e-predictor] data. Default is True. [e2e-predictor] :param _request_timeout: timeout setting for this request. If one [e2e-predictor] number provided, it will be total request [e2e-predictor] timeout. It can also be a pair (tuple) of [e2e-predictor] (connection, read) timeouts. [e2e-predictor] """ [e2e-predictor] method = method.upper() [e2e-predictor] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-predictor] 'PATCH', 'OPTIONS'] [e2e-predictor] [e2e-predictor] if post_params and body: [e2e-predictor] raise ApiValueError( [e2e-predictor] "body parameter cannot be used with post_params parameter." [e2e-predictor] ) [e2e-predictor] [e2e-predictor] post_params = post_params or {} [e2e-predictor] headers = headers or {} [e2e-predictor] [e2e-predictor] timeout = None [e2e-predictor] if _request_timeout: [e2e-predictor] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-predictor] timeout = urllib3.Timeout(total=_request_timeout) [e2e-predictor] elif (isinstance(_request_timeout, tuple) and [e2e-predictor] len(_request_timeout) == 2): [e2e-predictor] timeout = urllib3.Timeout( [e2e-predictor] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-predictor] [e2e-predictor] if 'Content-Type' not in headers: [e2e-predictor] headers['Content-Type'] = 'application/json' [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-predictor] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-predictor] if query_params: [e2e-predictor] url += '?' + urlencode(query_params) [e2e-predictor] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-predictor] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-predictor] if headers['Content-Type'] == 'application/json-patch+json': [e2e-predictor] if not isinstance(body, list): [e2e-predictor] headers['Content-Type'] = \ [e2e-predictor] 'application/strategic-merge-patch+json' [e2e-predictor] request_body = None [e2e-predictor] if body is not None: [e2e-predictor] request_body = json.dumps(body) [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] body=request_body, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] fields=post_params, [e2e-predictor] encode_multipart=False, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] elif headers['Content-Type'] == 'multipart/form-data': [e2e-predictor] # must del headers['Content-Type'], or the correct [e2e-predictor] # Content-Type which generated by urllib3 will be [e2e-predictor] # overwritten. [e2e-predictor] del headers['Content-Type'] [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] fields=post_params, [e2e-predictor] encode_multipart=True, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] # Pass a `string` parameter directly in the body to support [e2e-predictor] # other content types than Json when `body` argument is [e2e-predictor] # provided in serialized form [e2e-predictor] elif isinstance(body, str) or isinstance(body, bytes): [e2e-predictor] request_body = body [e2e-predictor] r = self.pool_manager.request( [e2e-predictor] method, url, [e2e-predictor] body=request_body, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] else: [e2e-predictor] # Cannot generate the request from given parameters [e2e-predictor] msg = """Cannot prepare a request message for provided [e2e-predictor] arguments. Please check that your arguments match [e2e-predictor] declared content type.""" [e2e-predictor] raise ApiException(status=0, reason=msg) [e2e-predictor] # For `GET`, `HEAD` [e2e-predictor] else: [e2e-predictor] > r = self.pool_manager.request(method, url, [e2e-predictor] fields=query_params, [e2e-predictor] preload_content=_preload_content, [e2e-predictor] timeout=timeout, [e2e-predictor] headers=headers) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:217: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None, fields = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] json = None, urlopen_kw = {'preload_content': True, 'timeout': None} [e2e-predictor] [e2e-predictor] def request( [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] fields: _TYPE_FIELDS | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] json: typing.Any | None = None, [e2e-predictor] **urlopen_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Make a request using :meth:`urlopen` with the appropriate encoding of [e2e-predictor] ``fields`` based on the ``method`` used. [e2e-predictor] [e2e-predictor] This is a convenience method that requires the least amount of manual [e2e-predictor] effort. It can be used in most situations, while still having the [e2e-predictor] option to drop down to more specific methods when necessary, such as [e2e-predictor] :meth:`request_encode_url`, :meth:`request_encode_body`, [e2e-predictor] or even the lowest level :meth:`urlopen`. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param fields: [e2e-predictor] Data to encode and send in the URL or request body, depending on ``method``. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param json: [e2e-predictor] Data to encode and send as JSON with UTF-encoded in the request body. [e2e-predictor] The ``"Content-Type"`` header will be set to ``"application/json"`` [e2e-predictor] unless specified otherwise. [e2e-predictor] """ [e2e-predictor] method = method.upper() [e2e-predictor] [e2e-predictor] if json is not None and body is not None: [e2e-predictor] raise TypeError( [e2e-predictor] "request got values for both 'body' and 'json' parameters which are mutually exclusive" [e2e-predictor] ) [e2e-predictor] [e2e-predictor] if json is not None: [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not ("content-type" in map(str.lower, headers.keys())): [e2e-predictor] headers = HTTPHeaderDict(headers) [e2e-predictor] headers["Content-Type"] = "application/json" [e2e-predictor] [e2e-predictor] body = _json.dumps(json, separators=(",", ":"), ensure_ascii=False).encode( [e2e-predictor] "utf-8" [e2e-predictor] ) [e2e-predictor] [e2e-predictor] if body is not None: [e2e-predictor] urlopen_kw["body"] = body [e2e-predictor] [e2e-predictor] if method in self._encode_url_methods: [e2e-predictor] > return self.request_encode_url( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] fields=fields, # type: ignore[arg-type] [e2e-predictor] headers=headers, [e2e-predictor] **urlopen_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/_request_methods.py:135: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] fields = [] [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] urlopen_kw = {'preload_content': True, 'timeout': None} [e2e-predictor] extra_kw = {'headers': {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'}, 'preload_content': True, 'timeout': None} [e2e-predictor] [e2e-predictor] def request_encode_url( [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] fields: _TYPE_ENCODE_URL_FIELDS | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] **urlopen_kw: str, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Make a request using :meth:`urlopen` with the ``fields`` encoded in [e2e-predictor] the url. This is useful for request methods like GET, HEAD, DELETE, etc. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param fields: [e2e-predictor] Data to encode and send in the URL. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] """ [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] extra_kw: dict[str, typing.Any] = {"headers": headers} [e2e-predictor] extra_kw.update(urlopen_kw) [e2e-predictor] [e2e-predictor] if fields: [e2e-predictor] url += "?" + urlencode(fields) [e2e-predictor] [e2e-predictor] > return self.urlopen(method, url, **extra_kw) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/_request_methods.py:182: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = 'https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] redirect = True [e2e-predictor] kw = {'assert_same_host': False, 'headers': {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'}, 'preload_content': True, 'redirect': False, ...} [e2e-predictor] u = Url(scheme='https', auth=None, host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443, path='/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways', query=None, fragment=None) [e2e-predictor] conn = [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, method: str, url: str, redirect: bool = True, **kw: typing.Any [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Same as :meth:`urllib3.HTTPConnectionPool.urlopen` [e2e-predictor] with custom cross-host redirect logic and only sends the request-uri [e2e-predictor] portion of the ``url``. [e2e-predictor] [e2e-predictor] The given ``url`` parameter must be absolute, such that an appropriate [e2e-predictor] :class:`urllib3.connectionpool.ConnectionPool` can be chosen for it. [e2e-predictor] """ [e2e-predictor] u = parse_url(url) [e2e-predictor] [e2e-predictor] if u.scheme is None: [e2e-predictor] warnings.warn( [e2e-predictor] "URLs without a scheme (ie 'https://') are deprecated and will raise an error " [e2e-predictor] "in urllib3 v3.0. To avoid this FutureWarning ensure all URLs " [e2e-predictor] "start with 'https://' or 'http://'. Read more in this issue: " [e2e-predictor] "https://github.com/urllib3/urllib3/issues/2920", [e2e-predictor] category=FutureWarning, [e2e-predictor] stacklevel=2, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme) [e2e-predictor] [e2e-predictor] kw["assert_same_host"] = False [e2e-predictor] kw["redirect"] = False [e2e-predictor] [e2e-predictor] if "headers" not in kw: [e2e-predictor] kw["headers"] = self.headers [e2e-predictor] [e2e-predictor] if self._proxy_requires_url_absolute_form(u): [e2e-predictor] response = conn.urlopen(method, url, **kw) [e2e-predictor] else: [e2e-predictor] > response = conn.urlopen(method, u.request_uri, **kw) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/poolmanager.py:457: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=2, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False [e2e-predictor] err = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] retries.sleep() [e2e-predictor] [e2e-predictor] # Keep track of the error for the retry warning. [e2e-predictor] err = e [e2e-predictor] [e2e-predictor] finally: [e2e-predictor] if not clean_exit: [e2e-predictor] # We hit some kind of exception, handled or otherwise. We need [e2e-predictor] # to throw the connection away unless explicitly told not to. [e2e-predictor] # Close the connection, set the variable to None, and make sure [e2e-predictor] # we put the None back in the pool to avoid leaking it. [e2e-predictor] if conn: [e2e-predictor] conn.close() [e2e-predictor] conn = None [e2e-predictor] release_this_conn = True [e2e-predictor] [e2e-predictor] if release_this_conn: [e2e-predictor] # Put the connection back to be reused. If the connection is [e2e-predictor] # expired then it will be None, which will get replaced with a [e2e-predictor] # fresh connection during _get_conn. [e2e-predictor] self._put_conn(conn) [e2e-predictor] [e2e-predictor] if not conn: [e2e-predictor] # Try again [e2e-predictor] log.warning( [e2e-predictor] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-predictor] ) [e2e-predictor] > return self.urlopen( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] body, [e2e-predictor] headers, [e2e-predictor] retries, [e2e-predictor] redirect, [e2e-predictor] assert_same_host, [e2e-predictor] timeout=timeout, [e2e-predictor] pool_timeout=pool_timeout, [e2e-predictor] release_conn=release_conn, [e2e-predictor] chunked=chunked, [e2e-predictor] body_pos=body_pos, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:872: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=1, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False [e2e-predictor] err = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] retries.sleep() [e2e-predictor] [e2e-predictor] # Keep track of the error for the retry warning. [e2e-predictor] err = e [e2e-predictor] [e2e-predictor] finally: [e2e-predictor] if not clean_exit: [e2e-predictor] # We hit some kind of exception, handled or otherwise. We need [e2e-predictor] # to throw the connection away unless explicitly told not to. [e2e-predictor] # Close the connection, set the variable to None, and make sure [e2e-predictor] # we put the None back in the pool to avoid leaking it. [e2e-predictor] if conn: [e2e-predictor] conn.close() [e2e-predictor] conn = None [e2e-predictor] release_this_conn = True [e2e-predictor] [e2e-predictor] if release_this_conn: [e2e-predictor] # Put the connection back to be reused. If the connection is [e2e-predictor] # expired then it will be None, which will get replaced with a [e2e-predictor] # fresh connection during _get_conn. [e2e-predictor] self._put_conn(conn) [e2e-predictor] [e2e-predictor] if not conn: [e2e-predictor] # Try again [e2e-predictor] log.warning( [e2e-predictor] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-predictor] ) [e2e-predictor] > return self.urlopen( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] body, [e2e-predictor] headers, [e2e-predictor] retries, [e2e-predictor] redirect, [e2e-predictor] assert_same_host, [e2e-predictor] timeout=timeout, [e2e-predictor] pool_timeout=pool_timeout, [e2e-predictor] release_conn=release_conn, [e2e-predictor] chunked=chunked, [e2e-predictor] body_pos=body_pos, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:872: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False [e2e-predictor] err = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] retries.sleep() [e2e-predictor] [e2e-predictor] # Keep track of the error for the retry warning. [e2e-predictor] err = e [e2e-predictor] [e2e-predictor] finally: [e2e-predictor] if not clean_exit: [e2e-predictor] # We hit some kind of exception, handled or otherwise. We need [e2e-predictor] # to throw the connection away unless explicitly told not to. [e2e-predictor] # Close the connection, set the variable to None, and make sure [e2e-predictor] # we put the None back in the pool to avoid leaking it. [e2e-predictor] if conn: [e2e-predictor] conn.close() [e2e-predictor] conn = None [e2e-predictor] release_this_conn = True [e2e-predictor] [e2e-predictor] if release_this_conn: [e2e-predictor] # Put the connection back to be reused. If the connection is [e2e-predictor] # expired then it will be None, which will get replaced with a [e2e-predictor] # fresh connection during _get_conn. [e2e-predictor] self._put_conn(conn) [e2e-predictor] [e2e-predictor] if not conn: [e2e-predictor] # Try again [e2e-predictor] log.warning( [e2e-predictor] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-predictor] ) [e2e-predictor] > return self.urlopen( [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] body, [e2e-predictor] headers, [e2e-predictor] retries, [e2e-predictor] redirect, [e2e-predictor] assert_same_host, [e2e-predictor] timeout=timeout, [e2e-predictor] pool_timeout=pool_timeout, [e2e-predictor] release_conn=release_conn, [e2e-predictor] chunked=chunked, [e2e-predictor] body_pos=body_pos, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:872: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] body = None [e2e-predictor] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-predictor] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] redirect = False, assert_same_host = False, timeout = None, pool_timeout = None [e2e-predictor] release_conn = True, chunked = False, body_pos = None, preload_content = True [e2e-predictor] decode_content = True, response_kw = {}, destination_scheme = None, conn = None [e2e-predictor] release_this_conn = True, http_tunnel_required = False, err = None [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] def urlopen( # type: ignore[override] [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: str, [e2e-predictor] body: _TYPE_BODY | None = None, [e2e-predictor] headers: typing.Mapping[str, str] | None = None, [e2e-predictor] retries: Retry | bool | int | None = None, [e2e-predictor] redirect: bool = True, [e2e-predictor] assert_same_host: bool = True, [e2e-predictor] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-predictor] pool_timeout: int | None = None, [e2e-predictor] release_conn: bool | None = None, [e2e-predictor] chunked: bool = False, [e2e-predictor] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-predictor] preload_content: bool = True, [e2e-predictor] decode_content: bool = True, [e2e-predictor] **response_kw: typing.Any, [e2e-predictor] ) -> BaseHTTPResponse: [e2e-predictor] """ [e2e-predictor] Get a connection from the pool and perform an HTTP request. This is the [e2e-predictor] lowest level call for making a request, so you'll need to specify all [e2e-predictor] the raw details. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] More commonly, it's appropriate to use a convenience method [e2e-predictor] such as :meth:`request`. [e2e-predictor] [e2e-predictor] .. note:: [e2e-predictor] [e2e-predictor] `release_conn` will only behave as expected if [e2e-predictor] `preload_content=False` because we want to make [e2e-predictor] `preload_content=False` the default behaviour someday soon without [e2e-predictor] breaking backwards compatibility. [e2e-predictor] [e2e-predictor] :param method: [e2e-predictor] HTTP request method (such as GET, POST, PUT, etc.) [e2e-predictor] [e2e-predictor] :param url: [e2e-predictor] The URL to perform the request on. [e2e-predictor] [e2e-predictor] :param body: [e2e-predictor] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-predictor] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-predictor] [e2e-predictor] :param headers: [e2e-predictor] Dictionary of custom headers to send, such as User-Agent, [e2e-predictor] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-predictor] these headers completely replace any pool-specific headers. [e2e-predictor] [e2e-predictor] :param retries: [e2e-predictor] Configure the number of retries to allow before raising a [e2e-predictor] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-predictor] [e2e-predictor] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-predictor] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-predictor] over different types of retries. [e2e-predictor] Pass an integer number to retry connection errors that many times, [e2e-predictor] but no other types of errors. Pass zero to never retry. [e2e-predictor] [e2e-predictor] If ``False``, then retries are disabled and any exception is raised [e2e-predictor] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-predictor] the redirect response will be returned. [e2e-predictor] [e2e-predictor] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-predictor] [e2e-predictor] :param redirect: [e2e-predictor] If True, automatically handle redirects (status codes 301, 302, [e2e-predictor] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-predictor] will disable redirect, too. [e2e-predictor] [e2e-predictor] :param assert_same_host: [e2e-predictor] If ``True``, will make sure that the host of the pool requests is [e2e-predictor] consistent else will raise HostChangedError. When ``False``, you can [e2e-predictor] use the pool on an HTTP proxy and request foreign hosts. [e2e-predictor] [e2e-predictor] :param timeout: [e2e-predictor] If specified, overrides the default timeout for this one [e2e-predictor] request. It may be a float (in seconds) or an instance of [e2e-predictor] :class:`urllib3.util.Timeout`. [e2e-predictor] [e2e-predictor] :param pool_timeout: [e2e-predictor] If set and the pool is set to block=True, then this method will [e2e-predictor] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-predictor] connection is available within the time period. [e2e-predictor] [e2e-predictor] :param bool preload_content: [e2e-predictor] If True, the response's body will be preloaded into memory. [e2e-predictor] [e2e-predictor] :param bool decode_content: [e2e-predictor] If True, will attempt to decode the body based on the [e2e-predictor] 'content-encoding' header. [e2e-predictor] [e2e-predictor] :param release_conn: [e2e-predictor] If False, then the urlopen call will not release the connection [e2e-predictor] back into the pool once a response is received (but will release if [e2e-predictor] you read the entire contents of the response such as when [e2e-predictor] `preload_content=True`). This is useful if you're not preloading [e2e-predictor] the response's content immediately. You will need to call [e2e-predictor] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-predictor] back into the pool. If None, it takes the value of ``preload_content`` [e2e-predictor] which defaults to ``True``. [e2e-predictor] [e2e-predictor] :param bool chunked: [e2e-predictor] If True, urllib3 will send the body using chunked transfer [e2e-predictor] encoding. Otherwise, urllib3 will send the body using the standard [e2e-predictor] content-length form. Defaults to False. [e2e-predictor] [e2e-predictor] :param int body_pos: [e2e-predictor] Position to seek to in file-like body in the event of a retry or [e2e-predictor] redirect. Typically this won't need to be set because urllib3 will [e2e-predictor] auto-populate the value when needed. [e2e-predictor] """ [e2e-predictor] # Ensure that the URL we're connecting to is properly encoded [e2e-predictor] if url.startswith("/"): [e2e-predictor] # URLs starting with / are inherently schemeless. [e2e-predictor] url = to_str(_encode_target(url)) [e2e-predictor] destination_scheme = None [e2e-predictor] else: [e2e-predictor] parsed_url = parse_url(url) [e2e-predictor] destination_scheme = parsed_url.scheme [e2e-predictor] url = to_str(parsed_url.url) [e2e-predictor] [e2e-predictor] if headers is None: [e2e-predictor] headers = self.headers [e2e-predictor] [e2e-predictor] if not isinstance(retries, Retry): [e2e-predictor] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-predictor] [e2e-predictor] if release_conn is None: [e2e-predictor] release_conn = preload_content [e2e-predictor] [e2e-predictor] # Check host [e2e-predictor] if assert_same_host and not self.is_same_host(url): [e2e-predictor] raise HostChangedError(self, url, retries) [e2e-predictor] [e2e-predictor] conn = None [e2e-predictor] [e2e-predictor] # Track whether `conn` needs to be released before [e2e-predictor] # returning/raising/recursing. Update this variable if necessary, and [e2e-predictor] # leave `release_conn` constant throughout the function. That way, if [e2e-predictor] # the function recurses, the original value of `release_conn` will be [e2e-predictor] # passed down into the recursive call, and its value will be respected. [e2e-predictor] # [e2e-predictor] # See issue #651 [1] for details. [e2e-predictor] # [e2e-predictor] # [1] [e2e-predictor] release_this_conn = release_conn [e2e-predictor] [e2e-predictor] http_tunnel_required = connection_requires_http_tunnel( [e2e-predictor] self.proxy, self.proxy_config, destination_scheme [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-predictor] # have to copy the headers dict so we can safely change it without those [e2e-predictor] # changes being reflected in anyone else's copy. [e2e-predictor] if not http_tunnel_required: [e2e-predictor] headers = headers.copy() # type: ignore[attr-defined] [e2e-predictor] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-predictor] [e2e-predictor] # Must keep the exception bound to a separate variable or else Python 3 [e2e-predictor] # complains about UnboundLocalError. [e2e-predictor] err = None [e2e-predictor] [e2e-predictor] # Keep track of whether we cleanly exited the except block. This [e2e-predictor] # ensures we do proper cleanup in finally. [e2e-predictor] clean_exit = False [e2e-predictor] [e2e-predictor] # Rewind body position, if needed. Record current position [e2e-predictor] # for future rewinds in the event of a redirect/retry. [e2e-predictor] body_pos = set_file_position(body, body_pos) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Request a connection from the queue. [e2e-predictor] timeout_obj = self._get_timeout(timeout) [e2e-predictor] conn = self._get_conn(timeout=pool_timeout) [e2e-predictor] [e2e-predictor] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-predictor] [e2e-predictor] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-predictor] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-predictor] try: [e2e-predictor] self._prepare_proxy(conn) [e2e-predictor] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-predictor] self._raise_timeout( [e2e-predictor] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-predictor] ) [e2e-predictor] raise [e2e-predictor] [e2e-predictor] # If we're going to release the connection in ``finally:``, then [e2e-predictor] # the response doesn't need to know about the connection. Otherwise [e2e-predictor] # it will also try to release it and we'll have a double-release [e2e-predictor] # mess. [e2e-predictor] response_conn = conn if not release_conn else None [e2e-predictor] [e2e-predictor] # Make the request on the HTTPConnection object [e2e-predictor] response = self._make_request( [e2e-predictor] conn, [e2e-predictor] method, [e2e-predictor] url, [e2e-predictor] timeout=timeout_obj, [e2e-predictor] body=body, [e2e-predictor] headers=headers, [e2e-predictor] chunked=chunked, [e2e-predictor] retries=retries, [e2e-predictor] response_conn=response_conn, [e2e-predictor] preload_content=preload_content, [e2e-predictor] decode_content=decode_content, [e2e-predictor] **response_kw, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] # Everything went great! [e2e-predictor] clean_exit = True [e2e-predictor] [e2e-predictor] except EmptyPoolError: [e2e-predictor] # Didn't get a connection from the pool, no need to clean up [e2e-predictor] clean_exit = True [e2e-predictor] release_this_conn = False [e2e-predictor] raise [e2e-predictor] [e2e-predictor] except ( [e2e-predictor] TimeoutError, [e2e-predictor] HTTPException, [e2e-predictor] OSError, [e2e-predictor] ProtocolError, [e2e-predictor] BaseSSLError, [e2e-predictor] SSLError, [e2e-predictor] CertificateError, [e2e-predictor] ProxyError, [e2e-predictor] ) as e: [e2e-predictor] # Discard the connection for these exceptions. It will be [e2e-predictor] # replaced during the next _get_conn() call. [e2e-predictor] clean_exit = False [e2e-predictor] new_e: Exception = e [e2e-predictor] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-predictor] new_e = SSLError(e) [e2e-predictor] if isinstance( [e2e-predictor] new_e, [e2e-predictor] ( [e2e-predictor] OSError, [e2e-predictor] NewConnectionError, [e2e-predictor] TimeoutError, [e2e-predictor] SSLError, [e2e-predictor] HTTPException, [e2e-predictor] ), [e2e-predictor] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-predictor] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-predictor] elif isinstance(new_e, (OSError, HTTPException)): [e2e-predictor] new_e = ProtocolError("Connection aborted.", new_e) [e2e-predictor] [e2e-predictor] > retries = retries.increment( [e2e-predictor] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:842: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-predictor] method = 'GET' [e2e-predictor] url = '/apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways' [e2e-predictor] response = None [e2e-predictor] error = NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.c...a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)") [e2e-predictor] _pool = [e2e-predictor] _stacktrace = [e2e-predictor] [e2e-predictor] def increment( [e2e-predictor] self, [e2e-predictor] method: str | None = None, [e2e-predictor] url: str | None = None, [e2e-predictor] response: BaseHTTPResponse | None = None, [e2e-predictor] error: Exception | None = None, [e2e-predictor] _pool: ConnectionPool | None = None, [e2e-predictor] _stacktrace: TracebackType | None = None, [e2e-predictor] ) -> Self: [e2e-predictor] """Return a new Retry object with incremented retry counters. [e2e-predictor] [e2e-predictor] :param response: A response object, or None, if the server did not [e2e-predictor] return a response. [e2e-predictor] :type response: :class:`~urllib3.response.BaseHTTPResponse` [e2e-predictor] :param Exception error: An error encountered during the request, or [e2e-predictor] None if the response was received successfully. [e2e-predictor] [e2e-predictor] :return: A new ``Retry`` object. [e2e-predictor] """ [e2e-predictor] if self.total is False and error: [e2e-predictor] # Disabled, indicate to re-raise the error. [e2e-predictor] raise reraise(type(error), error, _stacktrace) [e2e-predictor] [e2e-predictor] total = self.total [e2e-predictor] if total is not None: [e2e-predictor] total -= 1 [e2e-predictor] [e2e-predictor] connect = self.connect [e2e-predictor] read = self.read [e2e-predictor] redirect = self.redirect [e2e-predictor] status_count = self.status [e2e-predictor] other = self.other [e2e-predictor] cause = "unknown" [e2e-predictor] status = None [e2e-predictor] redirect_location = None [e2e-predictor] [e2e-predictor] if error and self._is_connection_error(error): [e2e-predictor] # Connect retry? [e2e-predictor] if connect is False: [e2e-predictor] raise reraise(type(error), error, _stacktrace) [e2e-predictor] elif connect is not None: [e2e-predictor] connect -= 1 [e2e-predictor] [e2e-predictor] elif error and self._is_read_error(error): [e2e-predictor] # Read retry? [e2e-predictor] if read is False or method is None or not self._is_method_retryable(method): [e2e-predictor] raise reraise(type(error), error, _stacktrace) [e2e-predictor] elif read is not None: [e2e-predictor] read -= 1 [e2e-predictor] [e2e-predictor] elif error: [e2e-predictor] # Other retry? [e2e-predictor] if other is not None: [e2e-predictor] other -= 1 [e2e-predictor] [e2e-predictor] elif response and response.get_redirect_location(): [e2e-predictor] # Redirect retry? [e2e-predictor] if redirect is not None: [e2e-predictor] redirect -= 1 [e2e-predictor] cause = "too many redirects" [e2e-predictor] response_redirect_location = response.get_redirect_location() [e2e-predictor] if response_redirect_location: [e2e-predictor] redirect_location = response_redirect_location [e2e-predictor] status = response.status [e2e-predictor] [e2e-predictor] else: [e2e-predictor] # Incrementing because of a server error like a 500 in [e2e-predictor] # status_forcelist and the given method is in the allowed_methods [e2e-predictor] cause = ResponseError.GENERIC_ERROR [e2e-predictor] if response and response.status: [e2e-predictor] if status_count is not None: [e2e-predictor] status_count -= 1 [e2e-predictor] cause = ResponseError.SPECIFIC_ERROR.format(status_code=response.status) [e2e-predictor] status = response.status [e2e-predictor] [e2e-predictor] history = self.history + ( [e2e-predictor] RequestHistory(method, url, error, status, redirect_location), [e2e-predictor] ) [e2e-predictor] [e2e-predictor] new_retry = self.new( [e2e-predictor] total=total, [e2e-predictor] connect=connect, [e2e-predictor] read=read, [e2e-predictor] redirect=redirect, [e2e-predictor] status=status_count, [e2e-predictor] other=other, [e2e-predictor] history=history, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] if new_retry.is_exhausted(): [e2e-predictor] reason = error or ResponseError(cause) [e2e-predictor] > raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] [e2e-predictor] E urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Max retries exceeded with url: /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways (Caused by NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/retry.py:543: MaxRetryError [e2e-predictor] ------------------------------ Captured log setup ------------------------------ [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/gateway.networking.k8s.io/v1/namespaces/kserve-ci-e2e-test/gateways [e2e-predictor] =================================== FAILURES =================================== [e2e-predictor] _________________________ test_sklearn_runtime_kserve __________________________ [e2e-predictor] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-predictor] [e2e-predictor] @contextlib.contextmanager [e2e-predictor] def map_httpcore_exceptions() -> typing.Iterator[None]: [e2e-predictor] try: [e2e-predictor] > yield [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpx/_transports/default.py:72: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] request = [e2e-predictor] [e2e-predictor] async def handle_async_request( [e2e-predictor] self, [e2e-predictor] request: Request, [e2e-predictor] ) -> Response: [e2e-predictor] assert isinstance(request.stream, AsyncByteStream) [e2e-predictor] [e2e-predictor] req = httpcore.Request( [e2e-predictor] method=request.method, [e2e-predictor] url=httpcore.URL( [e2e-predictor] scheme=request.url.raw_scheme, [e2e-predictor] host=request.url.raw_host, [e2e-predictor] port=request.url.port, [e2e-predictor] target=request.url.raw_path, [e2e-predictor] ), [e2e-predictor] headers=request.headers.raw, [e2e-predictor] content=request.stream, [e2e-predictor] extensions=request.extensions, [e2e-predictor] ) [e2e-predictor] with map_httpcore_exceptions(): [e2e-predictor] > resp = await self._pool.handle_async_request(req) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpx/_transports/default.py:377: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] request = [e2e-predictor] [e2e-predictor] async def handle_async_request(self, request: Request) -> Response: [e2e-predictor] """ [e2e-predictor] Send an HTTP request, and return an HTTP response. [e2e-predictor] [e2e-predictor] This is the core implementation that is called into by `.request()` or `.stream()`. [e2e-predictor] """ [e2e-predictor] scheme = request.url.scheme.decode() [e2e-predictor] if scheme == "": [e2e-predictor] raise UnsupportedProtocol( [e2e-predictor] "Request URL is missing an 'http://' or 'https://' protocol." [e2e-predictor] ) [e2e-predictor] if scheme not in ("http", "https", "ws", "wss"): [e2e-predictor] raise UnsupportedProtocol( [e2e-predictor] f"Request URL has an unsupported protocol '{scheme}://'." [e2e-predictor] ) [e2e-predictor] [e2e-predictor] timeouts = request.extensions.get("timeout", {}) [e2e-predictor] timeout = timeouts.get("pool", None) [e2e-predictor] [e2e-predictor] with self._optional_thread_lock: [e2e-predictor] # Add the incoming request to our request queue. [e2e-predictor] pool_request = AsyncPoolRequest(request) [e2e-predictor] self._requests.append(pool_request) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] while True: [e2e-predictor] with self._optional_thread_lock: [e2e-predictor] # Assign incoming requests to available connections, [e2e-predictor] # closing or creating new connections as required. [e2e-predictor] closing = self._assign_requests_to_connections() [e2e-predictor] await self._close_connections(closing) [e2e-predictor] [e2e-predictor] # Wait until this request has an assigned connection. [e2e-predictor] connection = await pool_request.wait_for_connection(timeout=timeout) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Send the request on the assigned connection. [e2e-predictor] response = await connection.handle_async_request( [e2e-predictor] pool_request.request [e2e-predictor] ) [e2e-predictor] except ConnectionNotAvailable: [e2e-predictor] # In some cases a connection may initially be available to [e2e-predictor] # handle a request, but then become unavailable. [e2e-predictor] # [e2e-predictor] # In this case we clear the connection and try again. [e2e-predictor] pool_request.clear_connection() [e2e-predictor] else: [e2e-predictor] break # pragma: nocover [e2e-predictor] [e2e-predictor] except BaseException as exc: [e2e-predictor] with self._optional_thread_lock: [e2e-predictor] # For any exception or cancellation we remove the request from [e2e-predictor] # the queue, and then re-assign requests to connections. [e2e-predictor] self._requests.remove(pool_request) [e2e-predictor] closing = self._assign_requests_to_connections() [e2e-predictor] [e2e-predictor] await self._close_connections(closing) [e2e-predictor] > raise exc from None [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpcore/_async/connection_pool.py:256: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] request = [e2e-predictor] [e2e-predictor] async def handle_async_request(self, request: Request) -> Response: [e2e-predictor] """ [e2e-predictor] Send an HTTP request, and return an HTTP response. [e2e-predictor] [e2e-predictor] This is the core implementation that is called into by `.request()` or `.stream()`. [e2e-predictor] """ [e2e-predictor] scheme = request.url.scheme.decode() [e2e-predictor] if scheme == "": [e2e-predictor] raise UnsupportedProtocol( [e2e-predictor] "Request URL is missing an 'http://' or 'https://' protocol." [e2e-predictor] ) [e2e-predictor] if scheme not in ("http", "https", "ws", "wss"): [e2e-predictor] raise UnsupportedProtocol( [e2e-predictor] f"Request URL has an unsupported protocol '{scheme}://'." [e2e-predictor] ) [e2e-predictor] [e2e-predictor] timeouts = request.extensions.get("timeout", {}) [e2e-predictor] timeout = timeouts.get("pool", None) [e2e-predictor] [e2e-predictor] with self._optional_thread_lock: [e2e-predictor] # Add the incoming request to our request queue. [e2e-predictor] pool_request = AsyncPoolRequest(request) [e2e-predictor] self._requests.append(pool_request) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] while True: [e2e-predictor] with self._optional_thread_lock: [e2e-predictor] # Assign incoming requests to available connections, [e2e-predictor] # closing or creating new connections as required. [e2e-predictor] closing = self._assign_requests_to_connections() [e2e-predictor] await self._close_connections(closing) [e2e-predictor] [e2e-predictor] # Wait until this request has an assigned connection. [e2e-predictor] connection = await pool_request.wait_for_connection(timeout=timeout) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] # Send the request on the assigned connection. [e2e-predictor] > response = await connection.handle_async_request( [e2e-predictor] pool_request.request [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpcore/_async/connection_pool.py:236: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = , request = [e2e-predictor] [e2e-predictor] async def handle_async_request(self, request: Request) -> Response: [e2e-predictor] if not self.can_handle_request(request.url.origin): [e2e-predictor] raise RuntimeError( [e2e-predictor] f"Attempted to send request to {request.url.origin} on connection to {self._origin}" [e2e-predictor] ) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] async with self._request_lock: [e2e-predictor] if self._connection is None: [e2e-predictor] stream = await self._connect(request) [e2e-predictor] [e2e-predictor] ssl_object = stream.get_extra_info("ssl_object") [e2e-predictor] http2_negotiated = ( [e2e-predictor] ssl_object is not None [e2e-predictor] and ssl_object.selected_alpn_protocol() == "h2" [e2e-predictor] ) [e2e-predictor] if http2_negotiated or (self._http2 and not self._http1): [e2e-predictor] from .http2 import AsyncHTTP2Connection [e2e-predictor] [e2e-predictor] self._connection = AsyncHTTP2Connection( [e2e-predictor] origin=self._origin, [e2e-predictor] stream=stream, [e2e-predictor] keepalive_expiry=self._keepalive_expiry, [e2e-predictor] ) [e2e-predictor] else: [e2e-predictor] self._connection = AsyncHTTP11Connection( [e2e-predictor] origin=self._origin, [e2e-predictor] stream=stream, [e2e-predictor] keepalive_expiry=self._keepalive_expiry, [e2e-predictor] ) [e2e-predictor] except BaseException as exc: [e2e-predictor] self._connect_failed = True [e2e-predictor] > raise exc [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpcore/_async/connection.py:101: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = , request = [e2e-predictor] [e2e-predictor] async def handle_async_request(self, request: Request) -> Response: [e2e-predictor] if not self.can_handle_request(request.url.origin): [e2e-predictor] raise RuntimeError( [e2e-predictor] f"Attempted to send request to {request.url.origin} on connection to {self._origin}" [e2e-predictor] ) [e2e-predictor] [e2e-predictor] try: [e2e-predictor] async with self._request_lock: [e2e-predictor] if self._connection is None: [e2e-predictor] > stream = await self._connect(request) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpcore/_async/connection.py:78: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = , request = [e2e-predictor] [e2e-predictor] async def _connect(self, request: Request) -> AsyncNetworkStream: [e2e-predictor] timeouts = request.extensions.get("timeout", {}) [e2e-predictor] sni_hostname = request.extensions.get("sni_hostname", None) [e2e-predictor] timeout = timeouts.get("connect", None) [e2e-predictor] [e2e-predictor] retries_left = self._retries [e2e-predictor] delays = exponential_backoff(factor=RETRIES_BACKOFF_FACTOR) [e2e-predictor] [e2e-predictor] while True: [e2e-predictor] try: [e2e-predictor] if self._uds is None: [e2e-predictor] kwargs = { [e2e-predictor] "host": self._origin.host.decode("ascii"), [e2e-predictor] "port": self._origin.port, [e2e-predictor] "local_address": self._local_address, [e2e-predictor] "timeout": timeout, [e2e-predictor] "socket_options": self._socket_options, [e2e-predictor] } [e2e-predictor] async with Trace("connect_tcp", logger, request, kwargs) as trace: [e2e-predictor] > stream = await self._network_backend.connect_tcp(**kwargs) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpcore/_async/connection.py:124: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] host = 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com' [e2e-predictor] port = 443, timeout = 180, local_address = None, socket_options = None [e2e-predictor] [e2e-predictor] async def connect_tcp( [e2e-predictor] self, [e2e-predictor] host: str, [e2e-predictor] port: int, [e2e-predictor] timeout: float | None = None, [e2e-predictor] local_address: str | None = None, [e2e-predictor] socket_options: typing.Iterable[SOCKET_OPTION] | None = None, [e2e-predictor] ) -> AsyncNetworkStream: [e2e-predictor] await self._init_backend() [e2e-predictor] > return await self._backend.connect_tcp( [e2e-predictor] host, [e2e-predictor] port, [e2e-predictor] timeout=timeout, [e2e-predictor] local_address=local_address, [e2e-predictor] socket_options=socket_options, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpcore/_backends/auto.py:31: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] host = 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com' [e2e-predictor] port = 443, timeout = 180, local_address = None, socket_options = [] [e2e-predictor] [e2e-predictor] async def connect_tcp( [e2e-predictor] self, [e2e-predictor] host: str, [e2e-predictor] port: int, [e2e-predictor] timeout: float | None = None, [e2e-predictor] local_address: str | None = None, [e2e-predictor] socket_options: typing.Iterable[SOCKET_OPTION] | None = None, [e2e-predictor] ) -> AsyncNetworkStream: # pragma: nocover [e2e-predictor] if socket_options is None: [e2e-predictor] socket_options = [] [e2e-predictor] exc_map = { [e2e-predictor] TimeoutError: ConnectTimeout, [e2e-predictor] OSError: ConnectError, [e2e-predictor] anyio.BrokenResourceError: ConnectError, [e2e-predictor] } [e2e-predictor] > with map_exceptions(exc_map): [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpcore/_backends/anyio.py:113: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] typ = [e2e-predictor] value = gaierror(-2, 'Name or service not known') [e2e-predictor] traceback = [e2e-predictor] [e2e-predictor] def __exit__(self, typ, value, traceback): [e2e-predictor] if typ is None: [e2e-predictor] try: [e2e-predictor] next(self.gen) [e2e-predictor] except StopIteration: [e2e-predictor] return False [e2e-predictor] else: [e2e-predictor] try: [e2e-predictor] raise RuntimeError("generator didn't stop") [e2e-predictor] finally: [e2e-predictor] self.gen.close() [e2e-predictor] else: [e2e-predictor] if value is None: [e2e-predictor] # Need to force instantiation so we can reliably [e2e-predictor] # tell if we get the same exception back [e2e-predictor] value = typ() [e2e-predictor] try: [e2e-predictor] > self.gen.throw(typ, value, traceback) [e2e-predictor] [e2e-predictor] /usr/lib64/python3.11/contextlib.py:158: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] map = {: , : , : } [e2e-predictor] [e2e-predictor] @contextlib.contextmanager [e2e-predictor] def map_exceptions(map: ExceptionMapping) -> typing.Iterator[None]: [e2e-predictor] try: [e2e-predictor] yield [e2e-predictor] except Exception as exc: # noqa: PIE786 [e2e-predictor] for from_exc, to_exc in map.items(): [e2e-predictor] if isinstance(exc, from_exc): [e2e-predictor] > raise to_exc(exc) from exc [e2e-predictor] E httpcore.ConnectError: [Errno -2] Name or service not known [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpcore/_exceptions.py:14: ConnectError [e2e-predictor] [e2e-predictor] The above exception was the direct cause of the following exception: [e2e-predictor] [e2e-predictor] rest_v1_client = [e2e-predictor] network_layer = 'openshift-route' [e2e-predictor] [e2e-predictor] @pytest.mark.predictor [e2e-predictor] @pytest.mark.kourier [e2e-predictor] @pytest.mark.asyncio(scope="session") [e2e-predictor] async def test_sklearn_runtime_kserve(rest_v1_client, network_layer): [e2e-predictor] service_name = "isvc-sklearn-runtime" [e2e-predictor] predictor = V1beta1PredictorSpec( [e2e-predictor] min_replicas=1, [e2e-predictor] model=V1beta1ModelSpec( [e2e-predictor] model_format=V1beta1ModelFormat( [e2e-predictor] name="sklearn", [e2e-predictor] ), [e2e-predictor] storage_uri="gs://kfserving-examples/models/sklearn/newsgroup/model.joblib", [e2e-predictor] resources=V1ResourceRequirements( [e2e-predictor] requests={"cpu": "2", "memory": "2Gi"}, [e2e-predictor] limits={"cpu": "2", "memory": "4Gi"}, [e2e-predictor] ), [e2e-predictor] args=["--workers=2"], [e2e-predictor] ), [e2e-predictor] ) [e2e-predictor] [e2e-predictor] isvc = V1beta1InferenceService( [e2e-predictor] api_version=constants.KSERVE_V1BETA1, [e2e-predictor] kind=constants.KSERVE_KIND_INFERENCESERVICE, [e2e-predictor] metadata=client.V1ObjectMeta( [e2e-predictor] name=service_name, [e2e-predictor] namespace=KSERVE_TEST_NAMESPACE, [e2e-predictor] labels={ [e2e-predictor] constants.KSERVE_LABEL_NETWORKING_VISIBILITY: constants.KSERVE_LABEL_NETWORKING_VISIBILITY_EXPOSED, [e2e-predictor] }, [e2e-predictor] ), [e2e-predictor] spec=V1beta1InferenceServiceSpec(predictor=predictor), [e2e-predictor] ) [e2e-predictor] [e2e-predictor] kserve_client = KServeClient( [e2e-predictor] config_file=os.environ.get("KUBECONFIG", "~/.kube/config") [e2e-predictor] ) [e2e-predictor] [e2e-predictor] kserve_client.create(isvc) [e2e-predictor] kserve_client.wait_isvc_ready(service_name, namespace=KSERVE_TEST_NAMESPACE) [e2e-predictor] tasks = [ [e2e-predictor] predict_isvc( [e2e-predictor] rest_v1_client, [e2e-predictor] service_name, [e2e-predictor] "./data/news_grouping_input_v1.json", [e2e-predictor] network_layer=network_layer, [e2e-predictor] ) [e2e-predictor] for _ in range(25) [e2e-predictor] ] [e2e-predictor] > responses = await asyncio.gather(*tasks) [e2e-predictor] [e2e-predictor] predictor/test_sklearn.py:203: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] client = [e2e-predictor] service_name = 'isvc-sklearn-runtime' [e2e-predictor] input = './data/news_grouping_input_v1.json', version = 'v1beta1' [e2e-predictor] model_name = 'isvc-sklearn-runtime', is_batch = False [e2e-predictor] network_layer = 'openshift-route', extra_headers = None [e2e-predictor] [e2e-predictor] async def predict_isvc( [e2e-predictor] client: InferenceRESTClient, [e2e-predictor] service_name, [e2e-predictor] input: Union[str, InferRequest], [e2e-predictor] version=constants.KSERVE_V1BETA1_VERSION, [e2e-predictor] model_name=None, [e2e-predictor] is_batch=False, [e2e-predictor] network_layer: str = "istio", [e2e-predictor] extra_headers: dict = None, [e2e-predictor] ) -> Union[InferResponse, Dict, List[Union[Dict, InferResponse]]]: [e2e-predictor] kfs_client = KServeClient( [e2e-predictor] config_file=os.environ.get("KUBECONFIG", "~/.kube/config") [e2e-predictor] ) [e2e-predictor] isvc = kfs_client.get( [e2e-predictor] service_name, [e2e-predictor] namespace=KSERVE_TEST_NAMESPACE, [e2e-predictor] version=version, [e2e-predictor] ) [e2e-predictor] scheme, cluster_ip, host, path = get_isvc_endpoint(isvc, network_layer) [e2e-predictor] if model_name is None: [e2e-predictor] model_name = service_name [e2e-predictor] base_url = f"{scheme}://{cluster_ip}{path}" [e2e-predictor] > return await predict( [e2e-predictor] client, [e2e-predictor] base_url, [e2e-predictor] host, [e2e-predictor] input, [e2e-predictor] model_name=model_name, [e2e-predictor] is_batch=is_batch, [e2e-predictor] is_graph=False, [e2e-predictor] extra_headers=extra_headers, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] common/utils.py:92: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] client = [e2e-predictor] url = 'https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com' [e2e-predictor] host = 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com' [e2e-predictor] input = './data/news_grouping_input_v1.json' [e2e-predictor] model_name = 'isvc-sklearn-runtime', is_batch = False, is_graph = False [e2e-predictor] extra_headers = None [e2e-predictor] [e2e-predictor] async def predict( [e2e-predictor] client: InferenceRESTClient, [e2e-predictor] url, [e2e-predictor] host, [e2e-predictor] input: Union[str, InferRequest], [e2e-predictor] model_name=None, [e2e-predictor] is_batch=False, [e2e-predictor] is_graph=False, [e2e-predictor] extra_headers: dict = None, [e2e-predictor] ) -> Union[InferResponse, Dict, List[Union[Dict, InferResponse]]]: [e2e-predictor] if isinstance(input, str): [e2e-predictor] with open(input) as json_file: [e2e-predictor] data = json.load(json_file) [e2e-predictor] else: [e2e-predictor] data = input [e2e-predictor] headers = {"Host": host, "Content-Type": "application/json"} [e2e-predictor] if extra_headers: [e2e-predictor] headers.update(extra_headers) [e2e-predictor] if is_batch: [e2e-predictor] with futures.ThreadPoolExecutor(max_workers=4) as executor: [e2e-predictor] loop = asyncio.get_running_loop() [e2e-predictor] future_list = [ [e2e-predictor] await loop.run_in_executor( [e2e-predictor] executor, [e2e-predictor] _predict, [e2e-predictor] client, [e2e-predictor] url, [e2e-predictor] input_data, [e2e-predictor] model_name, [e2e-predictor] headers, [e2e-predictor] is_graph, [e2e-predictor] ) [e2e-predictor] for input_data in data [e2e-predictor] ] [e2e-predictor] result = await asyncio.gather(*future_list) [e2e-predictor] else: [e2e-predictor] > result = await _predict( [e2e-predictor] client, [e2e-predictor] url, [e2e-predictor] data, [e2e-predictor] model_name=model_name, [e2e-predictor] headers=headers, [e2e-predictor] is_graph=is_graph, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] common/utils.py:140: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] client = [e2e-predictor] url = 'https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com' [e2e-predictor] input_data = {'inputs': ['Do what thou wilt shall be the whole of the Law. [Honestly.] The word of Sin is Restriction. [Would I kid... more in response to other posts in this thread. Invoke me under my stars. Love is the law, love under will. I am I!']} [e2e-predictor] model_name = 'isvc-sklearn-runtime' [e2e-predictor] headers = {'Content-Type': 'application/json', 'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com'} [e2e-predictor] is_graph = False [e2e-predictor] [e2e-predictor] async def _predict( [e2e-predictor] client: InferenceRESTClient, [e2e-predictor] url, [e2e-predictor] input_data, [e2e-predictor] model_name, [e2e-predictor] headers=None, [e2e-predictor] is_graph=False, [e2e-predictor] *, [e2e-predictor] retries: int = 5, [e2e-predictor] retry_delay: int = 5, [e2e-predictor] ) -> Union[InferResponse, Dict]: [e2e-predictor] transient_status_codes = {502, 503, 504} [e2e-predictor] logger.info("Sending Header = %s", headers) [e2e-predictor] logger.info("base url = %s", url) [e2e-predictor] for attempt in range(retries): [e2e-predictor] try: [e2e-predictor] > return await client.infer( [e2e-predictor] url, [e2e-predictor] input_data, [e2e-predictor] model_name=model_name, [e2e-predictor] headers=headers, [e2e-predictor] is_graph_endpoint=is_graph, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] common/utils.py:168: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] base_url = 'https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com' [e2e-predictor] data = b'{"inputs":["Do what thou wilt shall be the whole of the Law. [Honestly.] The word of Sin is Restriction. [Would I ki...more in response to other posts in this thread. Invoke me under my stars. Love is the law, love under will. I am I!"]}' [e2e-predictor] model_name = 'isvc-sklearn-runtime' [e2e-predictor] headers = {'Content-Type': 'application/json', 'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com'} [e2e-predictor] response_headers = None, is_graph_endpoint = False [e2e-predictor] timeout = [e2e-predictor] [e2e-predictor] async def infer( [e2e-predictor] self, [e2e-predictor] base_url: Union[httpx.URL, str], [e2e-predictor] data: Union[InferRequest, dict], [e2e-predictor] model_name: Optional[str] = None, [e2e-predictor] headers: Optional[Mapping[str, str]] = None, [e2e-predictor] response_headers: Dict[str, str] = None, [e2e-predictor] is_graph_endpoint: bool = False, [e2e-predictor] timeout: Union[float, None, tuple, httpx.Timeout] = httpx.USE_CLIENT_DEFAULT, [e2e-predictor] ) -> Union[InferResponse, Dict]: [e2e-predictor] """ [e2e-predictor] Run asynchronous inference using the supplied data. [e2e-predictor] :param base_url: Base url of the inference server. E.g. https://example.com:443, https://example.com:443/serving [e2e-predictor] :param data: Input data as InferRequest object. [e2e-predictor] :param model_name: (optional) Name of the model as string. If is_graph_endpoint is true this can be omitted. [e2e-predictor] If is_graph_endpoint is False and model_name is None, this will raise ValueError. [e2e-predictor] :param headers: (optional) HTTP headers to include when sending request. [e2e-predictor] :param is_graph_endpoint: (optional) If set to True the base_url will be considered as an inference graph [e2e-predictor] endpoint and will be used as it is for making the request regardless of the [e2e-predictor] protocol specified in the RESTConfig. The result will be returned as dict object. [e2e-predictor] Defaults to False. [e2e-predictor] :param timeout: (optional) The maximum end-to-end time, in seconds, the request is allowed to take. This will [e2e-predictor] override the timeout in the RESTConfig. The default value is 60 seconds. [e2e-predictor] To disable timeout explicitly set it to 'None'. [e2e-predictor] :return: Inference result as InferResponse object or python dict. [e2e-predictor] :raises HTTPStatusError for response codes other than 2xx. [e2e-predictor] :raises UnsupportedProtocol if the specified protocol version is not supported. [e2e-predictor] """ [e2e-predictor] if is_graph_endpoint: [e2e-predictor] url = base_url [e2e-predictor] elif model_name is None: [e2e-predictor] raise ValueError("model_name should not be 'None'") [e2e-predictor] elif is_v1(self._config.protocol): [e2e-predictor] url = self._construct_url( [e2e-predictor] base_url, f"{self._config.protocol}/models/{model_name}:predict" [e2e-predictor] ) [e2e-predictor] elif is_v2(self._config.protocol): [e2e-predictor] url = self._construct_url( [e2e-predictor] base_url, f"{self._config.protocol}/models/{model_name}/infer" [e2e-predictor] ) [e2e-predictor] else: [e2e-predictor] raise UnsupportedProtocol(self._config.protocol) [e2e-predictor] if self._config.verbose: [e2e-predictor] logger.info("url: %s", url) [e2e-predictor] logger.info("request data: %s", data) [e2e-predictor] if isinstance(data, InferRequest): [e2e-predictor] data, json_length = data.to_rest() [e2e-predictor] if json_length: [e2e-predictor] headers = headers or {} [e2e-predictor] headers[INFERENCE_CONTENT_LENGTH_HEADER] = str(json_length) [e2e-predictor] headers["content-type"] = "application/octet-stream" [e2e-predictor] if isinstance(data, dict): [e2e-predictor] data = orjson.dumps(data) [e2e-predictor] headers = dict(headers) if headers else {} [e2e-predictor] if not any(k.lower() == "content-type" for k in headers): [e2e-predictor] headers["content-type"] = "application/json" [e2e-predictor] > response = await self._client.post( [e2e-predictor] url, content=data, headers=headers, timeout=timeout [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/kserve/inference_client.py:503: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] url = URL('https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com/v1/models/isvc-sklearn-runtime:predict') [e2e-predictor] [e2e-predictor] async def post( [e2e-predictor] self, [e2e-predictor] url: URL | str, [e2e-predictor] *, [e2e-predictor] content: RequestContent | None = None, [e2e-predictor] data: RequestData | None = None, [e2e-predictor] files: RequestFiles | None = None, [e2e-predictor] json: typing.Any | None = None, [e2e-predictor] params: QueryParamTypes | None = None, [e2e-predictor] headers: HeaderTypes | None = None, [e2e-predictor] cookies: CookieTypes | None = None, [e2e-predictor] auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, [e2e-predictor] follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, [e2e-predictor] timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, [e2e-predictor] extensions: RequestExtensions | None = None, [e2e-predictor] ) -> Response: [e2e-predictor] """ [e2e-predictor] Send a `POST` request. [e2e-predictor] [e2e-predictor] **Parameters**: See `httpx.request`. [e2e-predictor] """ [e2e-predictor] > return await self.request( [e2e-predictor] "POST", [e2e-predictor] url, [e2e-predictor] content=content, [e2e-predictor] data=data, [e2e-predictor] files=files, [e2e-predictor] json=json, [e2e-predictor] params=params, [e2e-predictor] headers=headers, [e2e-predictor] cookies=cookies, [e2e-predictor] auth=auth, [e2e-predictor] follow_redirects=follow_redirects, [e2e-predictor] timeout=timeout, [e2e-predictor] extensions=extensions, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpx/_client.py:1905: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = , method = 'POST' [e2e-predictor] url = URL('https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com/v1/models/isvc-sklearn-runtime:predict') [e2e-predictor] [e2e-predictor] async def request( [e2e-predictor] self, [e2e-predictor] method: str, [e2e-predictor] url: URL | str, [e2e-predictor] *, [e2e-predictor] content: RequestContent | None = None, [e2e-predictor] data: RequestData | None = None, [e2e-predictor] files: RequestFiles | None = None, [e2e-predictor] json: typing.Any | None = None, [e2e-predictor] params: QueryParamTypes | None = None, [e2e-predictor] headers: HeaderTypes | None = None, [e2e-predictor] cookies: CookieTypes | None = None, [e2e-predictor] auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, [e2e-predictor] follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, [e2e-predictor] timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, [e2e-predictor] extensions: RequestExtensions | None = None, [e2e-predictor] ) -> Response: [e2e-predictor] """ [e2e-predictor] Build and send a request. [e2e-predictor] [e2e-predictor] Equivalent to: [e2e-predictor] [e2e-predictor] ```python [e2e-predictor] request = client.build_request(...) [e2e-predictor] response = await client.send(request, ...) [e2e-predictor] ``` [e2e-predictor] [e2e-predictor] See `AsyncClient.build_request()`, `AsyncClient.send()` [e2e-predictor] and [Merging of configuration][0] for how the various parameters [e2e-predictor] are merged with client-level configuration. [e2e-predictor] [e2e-predictor] [0]: /advanced/clients/#merging-of-configuration [e2e-predictor] """ [e2e-predictor] [e2e-predictor] if cookies is not None: # pragma: no cover [e2e-predictor] message = ( [e2e-predictor] "Setting per-request cookies=<...> is being deprecated, because " [e2e-predictor] "the expected behaviour on cookie persistence is ambiguous. Set " [e2e-predictor] "cookies directly on the client instance instead." [e2e-predictor] ) [e2e-predictor] warnings.warn(message, DeprecationWarning) [e2e-predictor] [e2e-predictor] request = self.build_request( [e2e-predictor] method=method, [e2e-predictor] url=url, [e2e-predictor] content=content, [e2e-predictor] data=data, [e2e-predictor] files=files, [e2e-predictor] json=json, [e2e-predictor] params=params, [e2e-predictor] headers=headers, [e2e-predictor] cookies=cookies, [e2e-predictor] timeout=timeout, [e2e-predictor] extensions=extensions, [e2e-predictor] ) [e2e-predictor] > return await self.send(request, auth=auth, follow_redirects=follow_redirects) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpx/_client.py:1585: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] request = [e2e-predictor] [e2e-predictor] async def send( [e2e-predictor] self, [e2e-predictor] request: Request, [e2e-predictor] *, [e2e-predictor] stream: bool = False, [e2e-predictor] auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, [e2e-predictor] follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, [e2e-predictor] ) -> Response: [e2e-predictor] """ [e2e-predictor] Send a request. [e2e-predictor] [e2e-predictor] The request is sent as-is, unmodified. [e2e-predictor] [e2e-predictor] Typically you'll want to build one with `AsyncClient.build_request()` [e2e-predictor] so that any client-level configuration is merged into the request, [e2e-predictor] but passing an explicit `httpx.Request()` is supported as well. [e2e-predictor] [e2e-predictor] See also: [Request instances][0] [e2e-predictor] [e2e-predictor] [0]: /advanced/clients/#request-instances [e2e-predictor] """ [e2e-predictor] if self._state == ClientState.CLOSED: [e2e-predictor] raise RuntimeError("Cannot send a request, as the client has been closed.") [e2e-predictor] [e2e-predictor] self._state = ClientState.OPENED [e2e-predictor] follow_redirects = ( [e2e-predictor] self.follow_redirects [e2e-predictor] if isinstance(follow_redirects, UseClientDefault) [e2e-predictor] else follow_redirects [e2e-predictor] ) [e2e-predictor] [e2e-predictor] self._set_timeout(request) [e2e-predictor] [e2e-predictor] auth = self._build_request_auth(request, auth) [e2e-predictor] [e2e-predictor] > response = await self._send_handling_auth( [e2e-predictor] request, [e2e-predictor] auth=auth, [e2e-predictor] follow_redirects=follow_redirects, [e2e-predictor] history=[], [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpx/_client.py:1674: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] request = [e2e-predictor] auth = , follow_redirects = False [e2e-predictor] history = [] [e2e-predictor] [e2e-predictor] async def _send_handling_auth( [e2e-predictor] self, [e2e-predictor] request: Request, [e2e-predictor] auth: Auth, [e2e-predictor] follow_redirects: bool, [e2e-predictor] history: list[Response], [e2e-predictor] ) -> Response: [e2e-predictor] auth_flow = auth.async_auth_flow(request) [e2e-predictor] try: [e2e-predictor] request = await auth_flow.__anext__() [e2e-predictor] [e2e-predictor] while True: [e2e-predictor] > response = await self._send_handling_redirects( [e2e-predictor] request, [e2e-predictor] follow_redirects=follow_redirects, [e2e-predictor] history=history, [e2e-predictor] ) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpx/_client.py:1702: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] request = [e2e-predictor] follow_redirects = False, history = [] [e2e-predictor] [e2e-predictor] async def _send_handling_redirects( [e2e-predictor] self, [e2e-predictor] request: Request, [e2e-predictor] follow_redirects: bool, [e2e-predictor] history: list[Response], [e2e-predictor] ) -> Response: [e2e-predictor] while True: [e2e-predictor] if len(history) > self.max_redirects: [e2e-predictor] raise TooManyRedirects( [e2e-predictor] "Exceeded maximum allowed redirects.", request=request [e2e-predictor] ) [e2e-predictor] [e2e-predictor] for hook in self._event_hooks["request"]: [e2e-predictor] await hook(request) [e2e-predictor] [e2e-predictor] > response = await self._send_single_request(request) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpx/_client.py:1739: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] request = [e2e-predictor] [e2e-predictor] async def _send_single_request(self, request: Request) -> Response: [e2e-predictor] """ [e2e-predictor] Sends a single request, without handling any redirections. [e2e-predictor] """ [e2e-predictor] transport = self._transport_for_url(request.url) [e2e-predictor] timer = Timer() [e2e-predictor] await timer.async_start() [e2e-predictor] [e2e-predictor] if not isinstance(request.stream, AsyncByteStream): [e2e-predictor] raise RuntimeError( [e2e-predictor] "Attempted to send an sync request with an AsyncClient instance." [e2e-predictor] ) [e2e-predictor] [e2e-predictor] with request_context(request=request): [e2e-predictor] > response = await transport.handle_async_request(request) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpx/_client.py:1776: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] request = [e2e-predictor] [e2e-predictor] async def handle_async_request(self, request: httpx.Request) -> httpx.Response: [e2e-predictor] """Sends an HTTP request, possibly with retries. [e2e-predictor] [e2e-predictor] Args: [e2e-predictor] request: The request to perform. [e2e-predictor] [e2e-predictor] Returns: [e2e-predictor] The final response. [e2e-predictor] """ [e2e-predictor] if self._async_transport is None: [e2e-predictor] raise RuntimeError("Async request received but no async transport available") [e2e-predictor] [e2e-predictor] logger.debug("handle_async_request started request=%s", request) [e2e-predictor] [e2e-predictor] if self.retry.is_retryable_method(request.method): [e2e-predictor] send_method = partial(self._async_transport.handle_async_request) [e2e-predictor] > response = await self._retry_operation_async(request, send_method) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpx_retries/transport.py:116: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] request = [e2e-predictor] send_method = functools.partial(>) [e2e-predictor] [e2e-predictor] async def _retry_operation_async( [e2e-predictor] self, [e2e-predictor] request: httpx.Request, [e2e-predictor] send_method: Callable[..., Coroutine[Any, Any, httpx.Response]], [e2e-predictor] ) -> httpx.Response: [e2e-predictor] retry = self.retry [e2e-predictor] response: Union[httpx.Response, Exception, None] = None [e2e-predictor] [e2e-predictor] while True: [e2e-predictor] if response is not None: [e2e-predictor] if isinstance(response, httpx.Response): [e2e-predictor] await response.aclose() [e2e-predictor] [e2e-predictor] logger.debug( [e2e-predictor] "_retry_operation_async retrying request=%s response=%s retry=%s", request, response, retry [e2e-predictor] ) [e2e-predictor] retry = retry.increment() [e2e-predictor] await retry.asleep(response) [e2e-predictor] try: [e2e-predictor] > response = await send_method(request) [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpx_retries/transport.py:171: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] request = [e2e-predictor] [e2e-predictor] async def handle_async_request( [e2e-predictor] self, [e2e-predictor] request: Request, [e2e-predictor] ) -> Response: [e2e-predictor] assert isinstance(request.stream, AsyncByteStream) [e2e-predictor] [e2e-predictor] req = httpcore.Request( [e2e-predictor] method=request.method, [e2e-predictor] url=httpcore.URL( [e2e-predictor] scheme=request.url.raw_scheme, [e2e-predictor] host=request.url.raw_host, [e2e-predictor] port=request.url.port, [e2e-predictor] target=request.url.raw_path, [e2e-predictor] ), [e2e-predictor] headers=request.headers.raw, [e2e-predictor] content=request.stream, [e2e-predictor] extensions=request.extensions, [e2e-predictor] ) [e2e-predictor] > with map_httpcore_exceptions(): [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpx/_transports/default.py:376: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] self = [e2e-predictor] typ = [e2e-predictor] value = ConnectError(gaierror(-2, 'Name or service not known')) [e2e-predictor] traceback = [e2e-predictor] [e2e-predictor] def __exit__(self, typ, value, traceback): [e2e-predictor] if typ is None: [e2e-predictor] try: [e2e-predictor] next(self.gen) [e2e-predictor] except StopIteration: [e2e-predictor] return False [e2e-predictor] else: [e2e-predictor] try: [e2e-predictor] raise RuntimeError("generator didn't stop") [e2e-predictor] finally: [e2e-predictor] self.gen.close() [e2e-predictor] else: [e2e-predictor] if value is None: [e2e-predictor] # Need to force instantiation so we can reliably [e2e-predictor] # tell if we get the same exception back [e2e-predictor] value = typ() [e2e-predictor] try: [e2e-predictor] > self.gen.throw(typ, value, traceback) [e2e-predictor] [e2e-predictor] /usr/lib64/python3.11/contextlib.py:158: [e2e-predictor] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-predictor] [e2e-predictor] @contextlib.contextmanager [e2e-predictor] def map_httpcore_exceptions() -> typing.Iterator[None]: [e2e-predictor] try: [e2e-predictor] yield [e2e-predictor] except Exception as exc: [e2e-predictor] mapped_exc = None [e2e-predictor] [e2e-predictor] for from_exc, to_exc in HTTPCORE_EXC_MAP.items(): [e2e-predictor] if not isinstance(exc, from_exc): [e2e-predictor] continue [e2e-predictor] # We want to map to the most specific exception we can find. [e2e-predictor] # Eg if `exc` is an `httpcore.ReadTimeout`, we want to map to [e2e-predictor] # `httpx.ReadTimeout`, not just `httpx.TimeoutException`. [e2e-predictor] if mapped_exc is None or issubclass(to_exc, mapped_exc): [e2e-predictor] mapped_exc = to_exc [e2e-predictor] [e2e-predictor] if mapped_exc is None: # pragma: no cover [e2e-predictor] raise [e2e-predictor] [e2e-predictor] message = str(exc) [e2e-predictor] > raise mapped_exc(message) from exc [e2e-predictor] E httpx.ConnectError: [Errno -2] Name or service not known [e2e-predictor] [e2e-predictor] ../../python/kserve/.venv/lib64/python3.11/site-packages/httpx/_transports/default.py:89: ConnectError [e2e-predictor] ------------------------------ Captured log call ------------------------------- [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:373 Host from isvc status URL = isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:374 Network layer = openshift-route [e2e-predictor] INFO kserve.trace:utils.py:377 Using external route host: isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] INFO kserve.trace:utils.py:164 Sending Header = {'Host': 'isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com', 'Content-Type': 'application/json'} [e2e-predictor] INFO kserve.trace:utils.py:165 base url = https://isvc-sklearn-runtime-kserve-ci-e2e-test.apps.ad5e3ef4-eead-4037-bc41-606a49dda1f3.prod.konfluxeaas.com [e2e-predictor] ---------------------------- Captured log teardown ----------------------------- [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/serving.kserve.io/v1beta1/namespaces/kserve-ci-e2e-test/inferenceservices/isvc-sklearn-runtime [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/serving.kserve.io/v1beta1/namespaces/kserve-ci-e2e-test/inferenceservices/isvc-sklearn-runtime [e2e-predictor] WARNING urllib3.connectionpool:connectionpool.py:869 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError("HTTPSConnection(host='a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com', port=6443): Failed to resolve 'a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com' ([Errno -2] Name or service not known)")': /apis/serving.kserve.io/v1beta1/namespaces/kserve-ci-e2e-test/inferenceservices/isvc-sklearn-runtime [e2e-predictor] =============================== warnings summary =============================== [e2e-predictor] predictor/test_canary_raw_deployment.py::test_canary_create [e2e-predictor] predictor/test_canary_raw_deployment.py:147: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-predictor] @pytest.mark.predictor [e2e-predictor] [e2e-predictor] predictor/test_canary_raw_deployment.py::test_canary_promote [e2e-predictor] predictor/test_canary_raw_deployment.py:194: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-predictor] @pytest.mark.predictor [e2e-predictor] [e2e-predictor] predictor/test_canary_raw_deployment.py::test_canary_rollback [e2e-predictor] predictor/test_canary_raw_deployment.py:256: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-predictor] @pytest.mark.predictor [e2e-predictor] [e2e-predictor] predictor/test_canary_raw_deployment.py::test_canary_force_stop [e2e-predictor] predictor/test_canary_raw_deployment.py:313: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-predictor] @pytest.mark.predictor [e2e-predictor] [e2e-predictor] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-predictor] - generated xml file: /workspace/artifacts-dir/junit_e2e-predictor_or_kserve_on_openshift.xml - [e2e-predictor] --------------------------------- JSON report ---------------------------------- [e2e-predictor] report saved to: /workspace/artifacts-dir/e2e_results-predictor_or_kserve_on_openshift.json [e2e-predictor] =========================== short test summary info ============================ [e2e-predictor] FAILED predictor/test_sklearn.py::test_sklearn_runtime_kserve - httpx.Connect... [e2e-predictor] ERROR predictor/test_sklearn.py::test_sklearn_v2_runtime_mlserver - urllib3.e... [e2e-predictor] ERROR predictor/test_sklearn.py::test_sklearn_v2 - urllib3.exceptions.MaxRetr... [e2e-predictor] ERROR predictor/test_sklearn.py::test_sklearn_v2_mixed - urllib3.exceptions.M... [e2e-predictor] ERROR predictor/test_tensorflow.py::test_tensorflow_kserve - urllib3.exceptio... [e2e-predictor] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 5 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! [e2e-predictor] !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! [e2e-predictor] = 1 failed, 36 passed, 14 skipped, 4 warnings, 4 errors in 6672.27s (1:51:12) == [must-gather] [must-gather ] OUT 2026-07-28T18:09:18.406735101Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] error getting cluster version: Get "https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/config.openshift.io/v1/clusterversions/version": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host [must-gather] ClusterID: [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Installing "" for : [must-gather] error getting cluster operators: Get "https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/config.openshift.io/v1/clusteroperators": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host [must-gather] ClusterOperators: [must-gather] clusteroperators are missing [must-gather] [must-gather] [must-gather] [must-gather] [must-gather] Error running must-gather collection: [must-gather] creating temp namespace: Post "https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host [must-gather] [must-gather] Falling back to `oc adm inspect clusterversion.v1.config.openshift.io,clusteroperators.v1.config.openshift.io` to collect basic cluster types. [must-gather] E0728 18:09:18.764400 23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api?timeout=32s\": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host" [must-gather] E0728 18:09:18.815262 23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api?timeout=32s\": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host" [must-gather] E0728 18:09:18.859344 23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api?timeout=32s\": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host" [must-gather] E0728 18:09:18.898889 23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api?timeout=32s\": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host" [must-gather] E0728 18:09:19.000037 23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api?timeout=32s\": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host" [must-gather] E0728 18:09:19.051306 23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api?timeout=32s\": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host" [must-gather] E0728 18:09:19.098855 23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api?timeout=32s\": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host" [must-gather] E0728 18:09:19.154365 23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api?timeout=32s\": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host" [must-gather] error completing cluster type inspection: error running backup collection: Get "https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api?timeout=32s": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host [must-gather] Falling back to `oc adm inspect namespace/openshift-cluster-version` to collect basic cluster named resources. [must-gather] E0728 18:09:19.209356 23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api?timeout=32s\": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host" [must-gather] E0728 18:09:19.259385 23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api?timeout=32s\": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host" [must-gather] E0728 18:09:19.328300 23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api?timeout=32s\": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host" [must-gather] E0728 18:09:19.364383 23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api?timeout=32s\": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host" [must-gather] error completing cluster named resource inspection: error running backup collection: Get "https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api?timeout=32s": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] error getting cluster version: Get "https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/config.openshift.io/v1/clusterversions/version": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host [must-gather] ClusterID: [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Installing "" for : [must-gather] error getting cluster operators: Get "https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/apis/config.openshift.io/v1/clusteroperators": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host [must-gather] ClusterOperators: [must-gather] clusteroperators are missing [must-gather] [must-gather] [must-gather] error: creating temp namespace: Post "https://a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces": dial tcp: lookup a43e7b912202b407493c74b3110e4330-dd4a0e6d7160a55c.elb.us-east-1.amazonaws.com on 172.30.0.10:53: no such host [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-hbdjx [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-hbdjx-e2e-predictor [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-hbdjx [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 154195 Jul 28 18:09 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-hbdjx/e2e-predictor.tar.gz [git-push-artifacts] [ci-artifacts 0446f7c] Updating CI Artifacts in e2e-predictor [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-hbdjx/e2e-predictor.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] 85517b5..0446f7c ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-28T18:09:29.412Z","type":3}]